Jul 24 22:14:52.517580 ip-10-0-139-136 systemd[1]: Starting Kubernetes Kubelet... Jul 24 22:14:52.927769 ip-10-0-139-136 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:52.927769 ip-10-0-139-136 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 22:14:52.927769 ip-10-0-139-136 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:52.927769 ip-10-0-139-136 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 22:14:52.927769 ip-10-0-139-136 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:52.930365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.930262 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 22:14:52.934557 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934542 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.934557 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934556 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934560 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934564 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934567 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934571 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934573 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934576 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934579 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934581 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934584 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934588 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934590 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934593 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934596 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934598 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934601 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934603 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934605 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934608 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934610 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.934618 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934613 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934615 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934618 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934620 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934623 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934625 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934628 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934632 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934634 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934637 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934640 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934642 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934644 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934647 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934649 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934651 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934655 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934659 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934662 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934665 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.935071 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934668 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934671 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934674 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934679 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934682 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934685 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934688 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934691 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934693 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934695 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934698 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934700 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934703 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934705 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934709 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934713 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934716 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934719 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934722 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.935582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934725 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934727 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934730 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934732 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934735 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934737 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934739 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934742 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934744 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934746 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934748 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934751 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934754 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934756 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934759 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934761 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934767 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934769 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934771 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.936028 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934774 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934776 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934779 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934782 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934784 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934786 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934789 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934791 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934795 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934798 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934801 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934803 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934805 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934808 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934810 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934813 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934815 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934817 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934820 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.934822 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.936510 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935538 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935546 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935549 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935552 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935555 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935558 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935560 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935563 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935565 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935568 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935571 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935573 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935576 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935578 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935581 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935583 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935586 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935588 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935591 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935594 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.936978 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935598 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935600 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935603 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935605 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935608 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935611 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935613 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935616 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935618 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935621 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935624 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935626 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935629 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935632 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935636 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935638 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935643 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935646 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935649 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.937489 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935652 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935655 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935657 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935660 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935663 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935665 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935668 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935670 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935673 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935675 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935677 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935680 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935683 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935685 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935688 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935691 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935693 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935696 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935698 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935700 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.937962 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935703 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935706 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935709 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935712 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935715 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935718 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935720 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935722 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935725 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935727 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935729 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935731 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935734 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935736 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935738 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935741 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935744 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935748 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935750 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935753 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.938461 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935755 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935759 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935762 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935764 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935766 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935769 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935771 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935773 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935776 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935778 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935780 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935783 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935785 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935788 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935790 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935794 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935796 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935799 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935801 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.935803 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.938934 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936186 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936191 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936194 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936197 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936200 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936202 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936205 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936208 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936210 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936213 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936216 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936219 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936221 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936224 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936226 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936229 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936231 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936234 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936236 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936239 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.939439 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936242 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936244 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936248 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936251 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936254 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936256 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936259 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936262 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936264 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936267 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936269 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936272 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936274 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936277 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936279 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936282 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936284 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936287 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936289 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936310 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.939896 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936314 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936318 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936323 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936328 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936331 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936334 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936336 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936338 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936341 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936343 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936346 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936348 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936351 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936353 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936356 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936358 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936361 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936363 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936365 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.940380 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936368 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936370 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936372 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936375 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936377 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936380 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936382 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936384 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936387 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936389 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936393 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936396 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936400 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936403 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936406 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936409 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936412 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936415 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936418 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936420 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.940902 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936423 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936426 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936429 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936431 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936433 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936444 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936447 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936449 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936452 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936454 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936456 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936459 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936463 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936466 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936470 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936490 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936493 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936495 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936498 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.941420 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.936500 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936568 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936575 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936582 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936586 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936592 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936596 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936600 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936605 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936608 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936615 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936619 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936622 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936625 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936628 2576 flags.go:64] FLAG: --cgroup-root="" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936631 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936634 2576 flags.go:64] FLAG: --client-ca-file="" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936637 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936640 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936644 2576 flags.go:64] FLAG: --cluster-domain="" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936646 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936649 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936652 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936655 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 22:14:52.941878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936659 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936662 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936665 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936668 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936671 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936674 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936677 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936680 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936683 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936687 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936690 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936693 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936696 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936699 2576 flags.go:64] FLAG: --enable-server="true" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936702 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936707 2576 flags.go:64] FLAG: --event-burst="100" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936710 2576 flags.go:64] FLAG: --event-qps="50" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936713 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936716 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936719 2576 flags.go:64] FLAG: --eviction-hard="" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936723 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936725 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936731 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936734 2576 flags.go:64] FLAG: --eviction-soft="" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936736 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 22:14:52.942474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936739 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936743 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936745 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936748 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936751 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936754 2576 flags.go:64] FLAG: --feature-gates="" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936757 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936760 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936763 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936766 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936769 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936772 2576 flags.go:64] FLAG: --help="false" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936775 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-136.ec2.internal" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936778 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936781 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936784 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936787 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936790 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936793 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936797 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936800 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936803 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936806 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936809 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 22:14:52.943056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936811 2576 flags.go:64] FLAG: --kube-reserved="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936814 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936817 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936820 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936823 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936827 2576 flags.go:64] FLAG: --lock-file="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936830 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936833 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936836 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936841 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936844 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936847 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936849 2576 flags.go:64] FLAG: --logging-format="text" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936852 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936855 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936858 2576 flags.go:64] FLAG: --manifest-url="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936861 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936865 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936868 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936872 2576 flags.go:64] FLAG: --max-pods="110" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936875 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936878 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936880 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936883 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936886 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 22:14:52.943634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936889 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936891 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936899 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936902 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936905 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936907 2576 flags.go:64] FLAG: --pod-cidr="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936910 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936915 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936918 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936921 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936923 2576 flags.go:64] FLAG: --port="10250" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936927 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936931 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d6c837c68bba8ca" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936934 2576 flags.go:64] FLAG: --qos-reserved="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936937 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936940 2576 flags.go:64] FLAG: --register-node="true" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936942 2576 flags.go:64] FLAG: --register-with-taints="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936946 2576 flags.go:64] FLAG: --registry-burst="10" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936949 2576 flags.go:64] FLAG: --registry-qps="5" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936951 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936954 2576 flags.go:64] FLAG: --reserved-memory="" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936958 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936961 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936964 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936967 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 22:14:52.944390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936969 2576 flags.go:64] FLAG: --runonce="false" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936972 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936975 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936978 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936981 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936984 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936986 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936990 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936993 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.936995 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937000 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937003 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937005 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937008 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937011 2576 flags.go:64] FLAG: --system-cgroups="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937013 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937019 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937021 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937024 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937029 2576 flags.go:64] FLAG: --tls-min-version="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937032 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937035 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937038 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937040 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937043 2576 flags.go:64] FLAG: --v="2" Jul 24 22:14:52.945230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937047 2576 flags.go:64] FLAG: --version="false" Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937050 2576 flags.go:64] FLAG: --vmodule="" Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937054 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.937057 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937172 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937176 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937178 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937181 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937184 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937187 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937189 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937192 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937194 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937197 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937199 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937202 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937204 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937208 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937210 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937213 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937215 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.945931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937218 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937220 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937222 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937225 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937227 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937231 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937234 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937237 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937239 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937241 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937244 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937246 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937248 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937251 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937254 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937256 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937259 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937261 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937263 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937266 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.946507 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937268 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937271 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937273 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937276 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937279 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937281 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937283 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937286 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937306 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937309 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937311 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937314 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937316 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937319 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937322 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937324 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937327 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937331 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937334 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937337 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.946981 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937339 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937342 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937344 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937347 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937349 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937352 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937354 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937356 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937359 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937361 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937364 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937366 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937369 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937371 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937373 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937376 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937380 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937383 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937386 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.947465 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937389 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937394 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937398 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937401 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937404 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937406 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937410 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937413 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937416 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937418 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937422 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937424 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937426 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937429 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937432 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937434 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937437 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937439 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937442 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.947913 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937444 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937446 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937449 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.937451 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.938092 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.946268 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.946283 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.946321 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.946328 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946386 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946390 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946393 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946396 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946399 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.948364 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946401 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946406 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946410 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946413 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946416 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946418 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946420 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946423 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946426 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946429 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946431 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946434 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946437 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946439 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946442 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946444 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946447 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946450 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946453 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946455 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.948731 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946458 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946460 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946463 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946465 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946468 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946470 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946473 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946475 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946478 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946481 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946485 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946488 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946491 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946493 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946496 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946498 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946501 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946503 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.949203 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946506 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946509 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946512 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946515 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946517 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946520 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946524 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946527 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946530 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946532 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946535 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946537 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946539 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946542 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946544 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946546 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946549 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946551 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946554 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946556 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.949774 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946559 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946561 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946563 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946566 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946568 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946571 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946573 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946575 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946578 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946580 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946582 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946585 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946587 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946589 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946591 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946594 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946597 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946599 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946602 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946604 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.950254 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946606 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946608 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946611 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946613 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946615 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946618 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946620 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946622 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946624 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946627 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946629 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946631 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946633 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946636 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946638 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.950754 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946640 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.946645 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946760 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946765 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946768 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946771 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946774 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946776 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946778 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946781 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946783 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946785 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946788 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946790 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946794 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:52.951103 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946796 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946799 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946801 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946803 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946806 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946808 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946810 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946813 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946815 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946818 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946820 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946822 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946825 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946827 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946829 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946832 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946834 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946837 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946839 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946841 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:52.951521 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946845 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946848 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946852 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946855 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946857 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946860 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946863 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946865 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946868 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946871 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946873 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946876 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946879 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946882 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946885 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946889 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946892 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946895 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946898 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:52.951990 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946900 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946903 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946905 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946907 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946910 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946912 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946914 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946917 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946919 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946922 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946924 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946926 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946929 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946931 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946934 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946936 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946938 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946941 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946943 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946946 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:52.952456 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946948 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946953 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946955 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946957 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946960 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946962 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946965 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946968 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946970 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946972 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946975 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946977 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946979 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946983 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946986 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946989 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946992 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946995 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.946998 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:52.952955 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947001 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947003 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947005 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947008 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947010 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947013 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947015 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:52.947017 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.947022 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.947130 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.950031 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.951326 2576 server.go:1019] "Starting client certificate rotation" Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.951421 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:14:52.953423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.951477 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:14:52.953917 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.953904 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:14:52.956728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.956711 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:14:52.965455 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.965438 2576 log.go:25] "Validated CRI v1 runtime API" Jul 24 22:14:52.969457 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.969440 2576 log.go:25] "Validated CRI v1 image API" Jul 24 22:14:52.970237 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.970221 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 22:14:52.972711 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.972690 2576 fs.go:135] Filesystem UUIDs: map[4182092c-352d-4adb-b40c-883866844584:/dev/nvme0n1p3 669cdfff-7d57-4b92-bc4a-14a0ce200c9d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 24 22:14:52.972786 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.972709 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 22:14:52.978373 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.978247 2576 manager.go:216] Machine: {Timestamp:2026-07-24 22:14:52.97694149 +0000 UTC m=+0.339380803 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3225342 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20161ab9680c058eb5e4d1e698c3f0 SystemUUID:ec20161a-b968-0c05-8eb5-e4d1e698c3f0 BootID:72a9d256-4101-4a1c-9da5-d17b9bc7ea1b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5e:99:a2:9a:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5e:99:a2:9a:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:5d:4b:ef:49:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 22:14:52.978373 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.978368 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 22:14:52.978472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.978449 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 22:14:52.981525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.981497 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 22:14:52.981664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.981528 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 22:14:52.982200 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.982190 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 22:14:52.982228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.982203 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 22:14:52.982228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.982210 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:14:52.982820 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.982809 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:14:52.983089 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.983074 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:14:52.984524 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.984229 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 22:14:52.987724 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987711 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:14:52.987836 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987827 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 22:14:52.987871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987856 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 24 22:14:52.987871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987866 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 22:14:52.987926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987879 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 22:14:52.987926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987887 2576 kubelet.go:398] "Adding apiserver pod source" Jul 24 22:14:52.987926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.987896 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 22:14:52.988212 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.988200 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:14:52.988251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.988221 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:14:52.990359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.990341 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 22:14:52.991540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.991526 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 22:14:52.991598 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.991565 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 22:14:52.992139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992126 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 22:14:52.992139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992142 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 22:14:52.992230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992148 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 22:14:52.992230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992154 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 22:14:52.992230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992161 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 22:14:52.992230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992166 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 22:14:52.992404 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992395 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 22:14:52.992525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992404 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 22:14:52.992525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992411 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 22:14:52.992525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992416 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 22:14:52.992525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992425 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 22:14:52.992525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.992446 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 22:14:52.994219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.994208 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 22:14:52.994257 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.994220 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 22:14:52.996398 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:52.996372 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 22:14:52.996398 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.996395 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 22:14:52.996591 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:52.996571 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 22:14:52.998067 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.998054 2576 server.go:1268] "Started kubelet" Jul 24 22:14:52.998311 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.998112 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 22:14:52.998311 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.998158 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 22:14:52.998311 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.998219 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 22:14:52.998960 ip-10-0-139-136 systemd[1]: Started Kubernetes Kubelet. Jul 24 22:14:52.999730 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:52.999711 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 22:14:53.000430 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.000408 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 24 22:14:53.003835 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.002740 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-136.ec2.internal.18c5598dd3a03bf6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-136.ec2.internal,UID:ip-10-0-139-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-136.ec2.internal,},FirstTimestamp:2026-07-24 22:14:52.998024182 +0000 UTC m=+0.360463493,LastTimestamp:2026-07-24 22:14:52.998024182 +0000 UTC m=+0.360463493,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-136.ec2.internal,}" Jul 24 22:14:53.004161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.004141 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfflp" Jul 24 22:14:53.004376 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.004359 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 22:14:53.006585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.006567 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 22:14:53.006659 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.006568 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 22:14:53.006710 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.006652 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.007332 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007315 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007336 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007319 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007427 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007439 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007481 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007494 2576 factory.go:55] Registering systemd factory Jul 24 22:14:53.007452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007504 2576 factory.go:223] Registration of the systemd container factory successfully Jul 24 22:14:53.007865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007729 2576 factory.go:153] Registering CRI-O factory Jul 24 22:14:53.007865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007740 2576 factory.go:223] Registration of the crio container factory successfully Jul 24 22:14:53.007865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007756 2576 factory.go:103] Registering Raw factory Jul 24 22:14:53.007865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.007764 2576 manager.go:1196] Started watching for new ooms in manager Jul 24 22:14:53.008383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.008372 2576 manager.go:349] Starting recovery of all containers Jul 24 22:14:53.008632 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.008609 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 22:14:53.008791 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.008769 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 22:14:53.011997 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.011974 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfflp" Jul 24 22:14:53.017976 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.017772 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 22:14:53.021483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.021460 2576 manager.go:354] Recovery completed Jul 24 22:14:53.025739 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.025725 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.027662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.027644 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.027727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.027680 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.027727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.027699 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.028247 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.028231 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 22:14:53.028247 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.028241 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.028370 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.028244 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 22:14:53.028370 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.028278 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:14:53.028992 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.028978 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.029036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.029004 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.029036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.029014 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.029412 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.029399 2576 policy_none.go:49] "None policy: Start" Jul 24 22:14:53.029468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.029419 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 22:14:53.029468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.029433 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 22:14:53.030416 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.030404 2576 policy_none.go:47] "Start" Jul 24 22:14:53.061973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.061954 2576 manager.go:337] "Starting Device Plugin manager" Jul 24 22:14:53.062106 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.061995 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 22:14:53.062106 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062008 2576 server.go:87] "Starting device plugin registration server" Jul 24 22:14:53.062352 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062339 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 22:14:53.062401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062354 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 22:14:53.062454 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062435 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 22:14:53.062543 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062527 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 22:14:53.062543 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.062541 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 22:14:53.063212 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.063196 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 22:14:53.063345 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.063242 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.162512 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.162472 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.163551 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.163457 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.163551 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.163492 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.163551 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.163507 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.163551 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.163544 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.171982 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.171958 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.172090 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.171991 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-136.ec2.internal\": node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.174837 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.174810 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 22:14:53.174947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.174849 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 22:14:53.174947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.174869 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 22:14:53.174947 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.174903 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 22:14:53.176689 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.176674 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:53.182067 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.182017 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.275027 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.274975 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal"] Jul 24 22:14:53.275189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.275053 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.276766 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.276751 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.276869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.276780 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.276869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.276791 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.278008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.277996 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.278118 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.278149 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278133 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.278707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278693 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.278797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278718 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.278797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278732 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.278797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278763 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.278797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278782 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.278797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.278791 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.279912 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.279899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.279948 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.279923 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:53.280639 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.280624 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:53.280735 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.280651 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:53.280735 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.280665 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:53.282066 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.282051 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.296914 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.296890 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-136.ec2.internal\" not found" node="ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.301335 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.301318 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-136.ec2.internal\" not found" node="ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.382617 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.382581 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.408624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.408589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.408624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.408626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.408831 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.408663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b42fb50bb861bf003dffd0219a78cbf-config\") pod \"kube-apiserver-proxy-ip-10-0-139-136.ec2.internal\" (UID: \"1b42fb50bb861bf003dffd0219a78cbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.483089 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.483007 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.509185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.509354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.509354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b42fb50bb861bf003dffd0219a78cbf-config\") pod \"kube-apiserver-proxy-ip-10-0-139-136.ec2.internal\" (UID: \"1b42fb50bb861bf003dffd0219a78cbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.509354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.509354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399eaf5b38805483cfd61cd6870bbb96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal\" (UID: \"399eaf5b38805483cfd61cd6870bbb96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.509493 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.509345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b42fb50bb861bf003dffd0219a78cbf-config\") pod \"kube-apiserver-proxy-ip-10-0-139-136.ec2.internal\" (UID: \"1b42fb50bb861bf003dffd0219a78cbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.583375 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.583333 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.599502 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.599478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.604829 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.604812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:53.683932 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.683893 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.784531 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.784441 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.885127 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.885088 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:53.950815 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.950788 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 22:14:53.951381 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:53.950921 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:14:53.985357 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:53.985319 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:54.016583 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.016550 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 22:09:53 +0000 UTC" deadline="2027-12-23 15:45:22.796857802 +0000 UTC" Jul 24 22:14:54.016716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.016589 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12401h30m28.780280969s" Jul 24 22:14:54.085598 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:54.085564 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:54.087314 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.087293 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:54.157411 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:54.157378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b42fb50bb861bf003dffd0219a78cbf.slice/crio-f1e982c396f7bbdf85ed6666bd33dca7921d729caa405e53f053b430832604bf WatchSource:0}: Error finding container f1e982c396f7bbdf85ed6666bd33dca7921d729caa405e53f053b430832604bf: Status 404 returned error can't find the container with id f1e982c396f7bbdf85ed6666bd33dca7921d729caa405e53f053b430832604bf Jul 24 22:14:54.157690 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:54.157668 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399eaf5b38805483cfd61cd6870bbb96.slice/crio-19f1f0a46925df94ca0bafcd1d49406f4ff67a522fd0c7a1b02710ed4385de9f WatchSource:0}: Error finding container 19f1f0a46925df94ca0bafcd1d49406f4ff67a522fd0c7a1b02710ed4385de9f: Status 404 returned error can't find the container with id 19f1f0a46925df94ca0bafcd1d49406f4ff67a522fd0c7a1b02710ed4385de9f Jul 24 22:14:54.162535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.162516 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:14:54.178445 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.178380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" event={"ID":"399eaf5b38805483cfd61cd6870bbb96","Type":"ContainerStarted","Data":"19f1f0a46925df94ca0bafcd1d49406f4ff67a522fd0c7a1b02710ed4385de9f"} Jul 24 22:14:54.179487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.179468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" event={"ID":"1b42fb50bb861bf003dffd0219a78cbf","Type":"ContainerStarted","Data":"f1e982c396f7bbdf85ed6666bd33dca7921d729caa405e53f053b430832604bf"} Jul 24 22:14:54.186645 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:54.186620 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:54.290405 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:54.290355 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-136.ec2.internal\" not found" Jul 24 22:14:54.302482 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.302455 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:54.339969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.339884 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:54.407580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.407543 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" Jul 24 22:14:54.419884 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.419864 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:14:54.421043 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.421028 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" Jul 24 22:14:54.430390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.430371 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:14:54.989473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.989442 2576 apiserver.go:52] "Watching apiserver" Jul 24 22:14:54.998199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.998167 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 22:14:54.999517 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:54.999486 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-kp6z4","openshift-multus/network-metrics-daemon-t42k6","openshift-network-operator/iptables-alerter-fnbln","openshift-ovn-kubernetes/ovnkube-node-2ffrc","kube-system/global-pull-secret-syncer-84ckx","kube-system/konnectivity-agent-6n9w6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal","openshift-multus/multus-additional-cni-plugins-8rkhl","openshift-network-diagnostics/network-check-target-gqshq","kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal","openshift-cluster-node-tuning-operator/tuned-r85jp","openshift-dns/node-resolver-rjcxr","openshift-image-registry/node-ca-h9tgc"] Jul 24 22:14:55.001178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.001156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.002658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.002322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.004220 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.003654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.006288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.006573 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.007035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v9qb\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.007970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008138 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008247 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008389 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dvp2c\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008426 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgvmh\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008722 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.008840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.009506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.009881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.010004 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.009948 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:14:55.011285 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.011064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.011915 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.011889 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5c2mb\"" Jul 24 22:14:55.012213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.011900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.012667 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.012646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.014029 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.014010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.015580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.015404 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.015580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.015505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.015580 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.015573 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:14:55.016537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.016501 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 22:14:55.016537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.016528 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 22:14:55.016787 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.016772 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 22:14:55.016866 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.016797 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2jwq2\"" Jul 24 22:14:55.016866 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.016841 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.017079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.017063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:55.017161 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.017131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:14:55.017161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.017142 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 22:14:55.017271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.017222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qfdvw\"" Jul 24 22:14:55.017271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.017235 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 22:14:55.017400 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.017289 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.019051 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.019031 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 22:14:55.019140 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.019083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 22:14:55.019956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.019913 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 22:14:55.020424 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-netd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-netns\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020599 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-bin\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c3974-1299-4e99-afd3-1c0691ad5d88-ovn-node-metrics-cert\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020731 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-log-socket\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020766 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3e650e-692a-4e51-910a-1772c02cb4bd-tmp-dir\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.020795 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020826 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-env-overrides\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020857 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-script-lib\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.020934 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-registration-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.021026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.021086 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48bbbb33-5bbe-4006-82af-7007a76c42a8-iptables-alerter-script\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.021138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-systemd-units\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.021279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspzr\" (UniqueName: \"kubernetes.io/projected/849c3974-1299-4e99-afd3-1c0691ad5d88-kube-api-access-sspzr\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.021478 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.021478 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021419 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-87bsr\"" Jul 24 22:14:55.021478 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-sys-fs\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.021811 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.020923 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 22:14:55.021985 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.021962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48bbbb33-5bbe-4006-82af-7007a76c42a8-host-slash\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.022091 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.022045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-config\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.022214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.022158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxjj\" (UniqueName: \"kubernetes.io/projected/6c3e650e-692a-4e51-910a-1772c02cb4bd-kube-api-access-zxxjj\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.022740 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.022721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.022957 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.022758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wdc\" (UniqueName: \"kubernetes.io/projected/977581cc-4ea1-4e86-994f-27963d068e47-kube-api-access-n7wdc\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.023017 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.022994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmr8\" (UniqueName: \"kubernetes.io/projected/48bbbb33-5bbe-4006-82af-7007a76c42a8-kube-api-access-szmr8\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.023070 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-slash\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023120 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-etc-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023223 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023210 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 22:14:55.023317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-ovn\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023361 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-node-log\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023406 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-socket-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.023456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-device-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.023456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c3e650e-692a-4e51-910a-1772c02cb4bd-hosts-file\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.023545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-kubelet\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.023646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-systemd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.023646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.023590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-var-lib-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.025917 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.025891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 22:14:55.026010 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.025920 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.026010 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.025935 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 22:14:55.026111 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.025891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.026228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.026211 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgp47\"" Jul 24 22:14:55.027462 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.027406 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 22:14:55.027657 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.027626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z7f8v\"" Jul 24 22:14:55.030496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.030472 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:14:55.109067 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.109029 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 22:14:55.124113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wdc\" (UniqueName: \"kubernetes.io/projected/977581cc-4ea1-4e86-994f-27963d068e47-kube-api-access-n7wdc\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.124331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-tmp\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.124331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-system-cni-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.124331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.124331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-cnibin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.124331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-netns\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-kubelet\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-ovn\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-socket-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-var-lib-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-ovn\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-var-lib-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.124575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-device-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c3e650e-692a-4e51-910a-1772c02cb4bd-hosts-file\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-socket-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxw98\" (UniqueName: \"kubernetes.io/projected/75481ec4-c026-4496-94fe-050f60955b2f-kube-api-access-mxw98\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-device-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-systemd\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c3e650e-692a-4e51-910a-1772c02cb4bd-hosts-file\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-host\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-netns\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-bin\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c3974-1299-4e99-afd3-1c0691ad5d88-ovn-node-metrics-cert\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-netns\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-bin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-log-socket\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3e650e-692a-4e51-910a-1772c02cb4bd-tmp-dir\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-bin\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-env-overrides\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-script-lib\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.124949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxr8\" (UniqueName: \"kubernetes.io/projected/afc8daf1-aced-453d-99be-31e6ca0a148d-kube-api-access-rhxr8\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-socket-dir-parent\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-k8s-cni-cncf-io\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-systemd-units\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspzr\" (UniqueName: \"kubernetes.io/projected/849c3974-1299-4e99-afd3-1c0691ad5d88-kube-api-access-sspzr\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-daemon-config\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95jd\" (UniqueName: \"kubernetes.io/projected/b5f35b73-7542-425f-b5b3-1316baa372d8-kube-api-access-h95jd\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75481ec4-c026-4496-94fe-050f60955b2f-host\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3e650e-692a-4e51-910a-1772c02cb4bd-tmp-dir\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-dbus\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125177 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 22:14:55.125848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-sys-fs\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-log-socket\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-sys-fs\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48bbbb33-5bbe-4006-82af-7007a76c42a8-host-slash\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-config\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-lib-modules\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnhm\" (UniqueName: \"kubernetes.io/projected/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-kube-api-access-6dnhm\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxjj\" (UniqueName: \"kubernetes.io/projected/6c3e650e-692a-4e51-910a-1772c02cb4bd-kube-api-access-zxxjj\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125579 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-env-overrides\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-systemd-units\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-os-release\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-system-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f7e2ba4-85bc-48de-af19-38e7bcababb4-agent-certs\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48bbbb33-5bbe-4006-82af-7007a76c42a8-host-slash\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e2ba4-85bc-48de-af19-38e7bcababb4-konnectivity-ca\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.126597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmr8\" (UniqueName: \"kubernetes.io/projected/48bbbb33-5bbe-4006-82af-7007a76c42a8-kube-api-access-szmr8\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-slash\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-script-lib\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-os-release\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-kubelet-config\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125976 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-slash\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.125982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-kubernetes\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-cnibin\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-etc-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-etc-openvswitch\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-node-log\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-kubelet\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-node-log\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-modprobe-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-systemd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-kubelet\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c3974-1299-4e99-afd3-1c0691ad5d88-ovnkube-config\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-netd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-run-systemd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126437 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c3974-1299-4e99-afd3-1c0691ad5d88-host-cni-netd\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-cni-binary-copy\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-conf-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysconfig\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-multus-certs\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwl5\" (UniqueName: \"kubernetes.io/projected/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-kube-api-access-zcwl5\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-conf\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-run\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75481ec4-c026-4496-94fe-050f60955b2f-serviceca\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-sys\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-registration-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48bbbb33-5bbe-4006-82af-7007a76c42a8-iptables-alerter-script\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.127945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-multus\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-hostroot\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977581cc-4ea1-4e86-994f-27963d068e47-registration-dir\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-etc-kubernetes\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.126985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-var-lib-kubelet\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.127008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-tuned\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.128756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.127278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48bbbb33-5bbe-4006-82af-7007a76c42a8-iptables-alerter-script\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.129263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.129056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c3974-1299-4e99-afd3-1c0691ad5d88-ovn-node-metrics-cert\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.134624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.134601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wdc\" (UniqueName: \"kubernetes.io/projected/977581cc-4ea1-4e86-994f-27963d068e47-kube-api-access-n7wdc\") pod \"aws-ebs-csi-driver-node-99p7d\" (UID: \"977581cc-4ea1-4e86-994f-27963d068e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.134914 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.134896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmr8\" (UniqueName: \"kubernetes.io/projected/48bbbb33-5bbe-4006-82af-7007a76c42a8-kube-api-access-szmr8\") pod \"iptables-alerter-fnbln\" (UID: \"48bbbb33-5bbe-4006-82af-7007a76c42a8\") " pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.135010 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.134985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspzr\" (UniqueName: \"kubernetes.io/projected/849c3974-1299-4e99-afd3-1c0691ad5d88-kube-api-access-sspzr\") pod \"ovnkube-node-2ffrc\" (UID: \"849c3974-1299-4e99-afd3-1c0691ad5d88\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.135137 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.135125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxjj\" (UniqueName: \"kubernetes.io/projected/6c3e650e-692a-4e51-910a-1772c02cb4bd-kube-api-access-zxxjj\") pod \"node-resolver-rjcxr\" (UID: \"6c3e650e-692a-4e51-910a-1772c02cb4bd\") " pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.156835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.156799 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-22psw" Jul 24 22:14:55.166230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.166192 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-22psw" Jul 24 22:14:55.227864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95jd\" (UniqueName: \"kubernetes.io/projected/b5f35b73-7542-425f-b5b3-1316baa372d8-kube-api-access-h95jd\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.227864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75481ec4-c026-4496-94fe-050f60955b2f-host\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.227864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-dbus\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-lib-modules\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75481ec4-c026-4496-94fe-050f60955b2f-host\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.227990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnhm\" (UniqueName: \"kubernetes.io/projected/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-kube-api-access-6dnhm\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-os-release\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-system-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f7e2ba4-85bc-48de-af19-38e7bcababb4-agent-certs\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e2ba4-85bc-48de-af19-38e7bcababb4-konnectivity-ca\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-dbus\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-os-release\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-kubelet-config\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-kubernetes\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-cnibin\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-lib-modules\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-modprobe-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-cni-binary-copy\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-os-release\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-conf-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0980d09d-775a-4f2a-82d9-7a5167813f50-kubelet-config\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysconfig\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-os-release\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-multus-certs\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwl5\" (UniqueName: \"kubernetes.io/projected/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-kube-api-access-zcwl5\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-cnibin\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-conf\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-modprobe-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-run\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.228859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-multus-certs\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-run\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75481ec4-c026-4496-94fe-050f60955b2f-serviceca\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-sys\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-multus\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-hostroot\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e2ba4-85bc-48de-af19-38e7bcababb4-konnectivity-ca\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-etc-kubernetes\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-var-lib-kubelet\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-tuned\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-tmp\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-system-cni-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.229764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75481ec4-c026-4496-94fe-050f60955b2f-serviceca\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-cnibin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-netns\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-kubelet\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-cnibin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxw98\" (UniqueName: \"kubernetes.io/projected/75481ec4-c026-4496-94fe-050f60955b2f-kube-api-access-mxw98\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-systemd\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-system-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-host\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-bin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxr8\" (UniqueName: \"kubernetes.io/projected/afc8daf1-aced-453d-99be-31e6ca0a148d-kube-api-access-rhxr8\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-socket-dir-parent\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-k8s-cni-cncf-io\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-daemon-config\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-cni-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-etc-kubernetes\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-d\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.230715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-sys\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-multus\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysctl-conf\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-cni-binary-copy\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-var-lib-kubelet\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-hostroot\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-daemon-config\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afc8daf1-aced-453d-99be-31e6ca0a148d-system-cni-dir\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.229887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-conf-dir\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.229971 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.230043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.730023149 +0000 UTC m=+3.092462467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-sysconfig\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.228449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-kubernetes\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-netns\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-kubelet\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-systemd\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.231605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-host\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-var-lib-cni-bin\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-multus-socket-dir-parent\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.230726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5f35b73-7542-425f-b5b3-1316baa372d8-host-run-k8s-cni-cncf-io\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.230876 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.230930 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.730914161 +0000 UTC m=+3.093353463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.232480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.231124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afc8daf1-aced-453d-99be-31e6ca0a148d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.232885 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.232827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-etc-tuned\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.232885 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.232835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f7e2ba4-85bc-48de-af19-38e7bcababb4-agent-certs\") pod \"konnectivity-agent-6n9w6\" (UID: \"3f7e2ba4-85bc-48de-af19-38e7bcababb4\") " pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.234173 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.234155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-tmp\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.237897 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.237877 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:55.237897 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.237900 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:55.238108 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.237915 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:55.238108 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.237992 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.737974219 +0000 UTC m=+3.100413519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:55.240447 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.240290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95jd\" (UniqueName: \"kubernetes.io/projected/b5f35b73-7542-425f-b5b3-1316baa372d8-kube-api-access-h95jd\") pod \"multus-kp6z4\" (UID: \"b5f35b73-7542-425f-b5b3-1316baa372d8\") " pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.240447 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.240378 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwl5\" (UniqueName: \"kubernetes.io/projected/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-kube-api-access-zcwl5\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.241540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.241521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnhm\" (UniqueName: \"kubernetes.io/projected/231e3285-b74a-4a4a-aa5f-0c507a0d9a3a-kube-api-access-6dnhm\") pod \"tuned-r85jp\" (UID: \"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a\") " pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.242547 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.242526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxw98\" (UniqueName: \"kubernetes.io/projected/75481ec4-c026-4496-94fe-050f60955b2f-kube-api-access-mxw98\") pod \"node-ca-h9tgc\" (UID: \"75481ec4-c026-4496-94fe-050f60955b2f\") " pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.243262 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.243243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxr8\" (UniqueName: \"kubernetes.io/projected/afc8daf1-aced-453d-99be-31e6ca0a148d-kube-api-access-rhxr8\") pod \"multus-additional-cni-plugins-8rkhl\" (UID: \"afc8daf1-aced-453d-99be-31e6ca0a148d\") " pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.276449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.276412 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:55.312040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.312006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" Jul 24 22:14:55.326830 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.326793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjcxr" Jul 24 22:14:55.336897 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.336866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fnbln" Jul 24 22:14:55.347835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.347814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:14:55.355791 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.355764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:14:55.362540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.362509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" Jul 24 22:14:55.370248 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.370223 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kp6z4" Jul 24 22:14:55.377942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.377911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h9tgc" Jul 24 22:14:55.384768 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.384745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r85jp" Jul 24 22:14:55.732680 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.732593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:55.732680 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.732632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:55.732894 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.732766 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.732894 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.732823 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:56.732805524 +0000 UTC m=+4.095244824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.733019 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.732996 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.733087 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.733076 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:14:56.733057749 +0000 UTC m=+4.095497062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.833762 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:55.833732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:55.833933 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.833900 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:55.833933 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.833919 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:55.833933 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.833929 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:55.834070 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:55.833985 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:56.833969148 +0000 UTC m=+4.196408446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:56.083730 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.083707 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849c3974_1299_4e99_afd3_1c0691ad5d88.slice/crio-252f7922466d2be18b3951c2ff6138a9cf657e651c9a4ca94a99394c6026e01c WatchSource:0}: Error finding container 252f7922466d2be18b3951c2ff6138a9cf657e651c9a4ca94a99394c6026e01c: Status 404 returned error can't find the container with id 252f7922466d2be18b3951c2ff6138a9cf657e651c9a4ca94a99394c6026e01c Jul 24 22:14:56.085317 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.084526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e2ba4_85bc_48de_af19_38e7bcababb4.slice/crio-e2553622889ad196e42b277e9878b80009a060becf2902191665b39e5acea208 WatchSource:0}: Error finding container e2553622889ad196e42b277e9878b80009a060becf2902191665b39e5acea208: Status 404 returned error can't find the container with id e2553622889ad196e42b277e9878b80009a060becf2902191665b39e5acea208 Jul 24 22:14:56.085317 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.085269 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc8daf1_aced_453d_99be_31e6ca0a148d.slice/crio-f0d18199fa3ec9117485b9489b6032dcc4ea8130155555f2ca868c82744558f9 WatchSource:0}: Error finding container f0d18199fa3ec9117485b9489b6032dcc4ea8130155555f2ca868c82744558f9: Status 404 returned error can't find the container with id f0d18199fa3ec9117485b9489b6032dcc4ea8130155555f2ca868c82744558f9 Jul 24 22:14:56.086329 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.086133 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231e3285_b74a_4a4a_aa5f_0c507a0d9a3a.slice/crio-71fc1bea91d3d6bcd716c3e8e69326c0f488280cbbc7b089dbb88a7660963417 WatchSource:0}: Error finding container 71fc1bea91d3d6bcd716c3e8e69326c0f488280cbbc7b089dbb88a7660963417: Status 404 returned error can't find the container with id 71fc1bea91d3d6bcd716c3e8e69326c0f488280cbbc7b089dbb88a7660963417 Jul 24 22:14:56.087239 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.087216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f35b73_7542_425f_b5b3_1316baa372d8.slice/crio-31d2495f0a40b6973fcd6900fe6273af3984f5dc87bef92ce692d564e465ec2b WatchSource:0}: Error finding container 31d2495f0a40b6973fcd6900fe6273af3984f5dc87bef92ce692d564e465ec2b: Status 404 returned error can't find the container with id 31d2495f0a40b6973fcd6900fe6273af3984f5dc87bef92ce692d564e465ec2b Jul 24 22:14:56.096241 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:14:56.096216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75481ec4_c026_4496_94fe_050f60955b2f.slice/crio-f4fe0ccb9c81ce0d552f6d8ba2bb340ff232b1a22c74c55c9c80f4f84ae03cc7 WatchSource:0}: Error finding container f4fe0ccb9c81ce0d552f6d8ba2bb340ff232b1a22c74c55c9c80f4f84ae03cc7: Status 404 returned error can't find the container with id f4fe0ccb9c81ce0d552f6d8ba2bb340ff232b1a22c74c55c9c80f4f84ae03cc7 Jul 24 22:14:56.170100 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.170055 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:09:55 +0000 UTC" deadline="2028-02-17 08:14:55.995276623 +0000 UTC" Jul 24 22:14:56.170100 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.170098 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13737h59m59.825181271s" Jul 24 22:14:56.175831 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.175810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:56.175947 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.175912 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:14:56.183663 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.183631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kp6z4" event={"ID":"b5f35b73-7542-425f-b5b3-1316baa372d8","Type":"ContainerStarted","Data":"31d2495f0a40b6973fcd6900fe6273af3984f5dc87bef92ce692d564e465ec2b"} Jul 24 22:14:56.185537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.185514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r85jp" event={"ID":"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a","Type":"ContainerStarted","Data":"71fc1bea91d3d6bcd716c3e8e69326c0f488280cbbc7b089dbb88a7660963417"} Jul 24 22:14:56.186825 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.186804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"252f7922466d2be18b3951c2ff6138a9cf657e651c9a4ca94a99394c6026e01c"} Jul 24 22:14:56.189587 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.189565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h9tgc" event={"ID":"75481ec4-c026-4496-94fe-050f60955b2f","Type":"ContainerStarted","Data":"f4fe0ccb9c81ce0d552f6d8ba2bb340ff232b1a22c74c55c9c80f4f84ae03cc7"} Jul 24 22:14:56.192040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.192007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fnbln" event={"ID":"48bbbb33-5bbe-4006-82af-7007a76c42a8","Type":"ContainerStarted","Data":"a8094c643ffb7b6fac70361eb10255eb9ffb5cfafd6728ae1a8ffd124596006d"} Jul 24 22:14:56.193063 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.193029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" event={"ID":"977581cc-4ea1-4e86-994f-27963d068e47","Type":"ContainerStarted","Data":"1025f24ac8263f38b392441349951577fb3ec79d529ea2f246bc66d26b09b7fb"} Jul 24 22:14:56.198351 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.198286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6n9w6" event={"ID":"3f7e2ba4-85bc-48de-af19-38e7bcababb4","Type":"ContainerStarted","Data":"e2553622889ad196e42b277e9878b80009a060becf2902191665b39e5acea208"} Jul 24 22:14:56.199263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.199240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerStarted","Data":"f0d18199fa3ec9117485b9489b6032dcc4ea8130155555f2ca868c82744558f9"} Jul 24 22:14:56.201127 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.201108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" event={"ID":"1b42fb50bb861bf003dffd0219a78cbf","Type":"ContainerStarted","Data":"7d3124084ed3aa7ae12d964066368f1a2f14b80e43b9be34412d4fe63c589504"} Jul 24 22:14:56.202018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.202001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjcxr" event={"ID":"6c3e650e-692a-4e51-910a-1772c02cb4bd","Type":"ContainerStarted","Data":"449904e80e0078532299c2e9190dd929ad97bc29c7739c0dd5e0e77b686d0b30"} Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.741509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.741554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.741697 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.741752 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:58.741736345 +0000 UTC m=+6.104175643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.742123 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:56.742206 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.742173 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:14:58.742156765 +0000 UTC m=+6.104596066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:56.842681 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:56.842069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:56.842681 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.842257 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:56.842681 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.842274 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:56.842681 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.842285 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:56.842681 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:56.842355 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:58.842337223 +0000 UTC m=+6.204776525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:57.171092 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.170995 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:09:55 +0000 UTC" deadline="2028-04-23 01:10:48.424741844 +0000 UTC" Jul 24 22:14:57.171092 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.171038 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15314h55m51.253708519s" Jul 24 22:14:57.175413 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.175387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:57.175546 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:57.175522 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:14:57.175921 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.175905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:57.176030 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:57.176011 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:14:57.214919 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.214880 2576 generic.go:356] "Generic (PLEG): container finished" podID="399eaf5b38805483cfd61cd6870bbb96" containerID="52b6b6ca014dec962062ec5dee6aca2d24cb224714298a5f7aa34000e937e07d" exitCode=0 Jul 24 22:14:57.215096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.215036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" event={"ID":"399eaf5b38805483cfd61cd6870bbb96","Type":"ContainerDied","Data":"52b6b6ca014dec962062ec5dee6aca2d24cb224714298a5f7aa34000e937e07d"} Jul 24 22:14:57.230366 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:57.230317 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-136.ec2.internal" podStartSLOduration=3.2302797930000002 podStartE2EDuration="3.230279793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:14:56.218094461 +0000 UTC m=+3.580533780" watchObservedRunningTime="2026-07-24 22:14:57.230279793 +0000 UTC m=+4.592719107" Jul 24 22:14:58.178867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.178462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:58.178867 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.178591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:14:58.179395 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.178974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:58.179395 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.179073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:14:58.227510 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.227430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" event={"ID":"399eaf5b38805483cfd61cd6870bbb96","Type":"ContainerStarted","Data":"afc33b30a01543b23963d2e20c0a502e18200414f84300e0762b70fda9179159"} Jul 24 22:14:58.757206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.757167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:58.757402 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.757216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:14:58.757402 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.757351 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:58.757402 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.757362 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:58.757584 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.757412 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:02.757392019 +0000 UTC m=+10.119831333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:58.757584 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.757431 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:02.757421773 +0000 UTC m=+10.119861071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:58.857679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:58.857636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:14:58.857862 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.857829 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:58.857862 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.857849 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:58.857862 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.857860 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:58.858016 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:58.857914 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:02.857896114 +0000 UTC m=+10.220335434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:59.179952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:14:59.179852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:14:59.180407 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:14:59.180010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:00.176040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:00.176015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:00.176197 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:00.176118 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:00.176426 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:00.176411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:00.176569 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:00.176488 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:01.180063 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:01.180035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:01.180499 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:01.180170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:02.175517 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:02.175482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:02.175683 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.175613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:02.176126 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:02.175991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:02.176126 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.176082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:02.789425 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:02.789380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:02.789425 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:02.789434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:02.789941 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.789555 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:02.789941 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.789558 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:02.789941 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.789640 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:10.789618741 +0000 UTC m=+18.152058062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:02.789941 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.789675 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:10.789658228 +0000 UTC m=+18.152097528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:02.890618 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:02.890580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:02.890819 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.890783 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:15:02.890819 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.890815 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:15:02.890948 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.890829 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:02.890948 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:02.890891 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:10.890872481 +0000 UTC m=+18.253311787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:03.183640 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:03.183565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:03.185356 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:03.185258 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:04.175499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:04.175469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:04.175865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:04.175469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:04.175865 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:04.175730 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:04.175865 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:04.175602 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:05.177047 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:05.176009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:05.177473 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:05.177169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:06.175837 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:06.175807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:06.175837 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:06.175840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:06.176081 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:06.175944 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:06.176081 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:06.176018 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:07.175566 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:07.175526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:07.176043 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:07.175687 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:08.175900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:08.175852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:08.176312 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:08.175852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:08.176312 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:08.176018 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:08.176312 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:08.176058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:09.175548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:09.175503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:09.175742 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:09.175660 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:10.176090 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:10.176064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:10.176464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:10.176073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:10.176464 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.176156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:10.176464 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.176214 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:10.855057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:10.855018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:10.855253 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:10.855063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:10.855253 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.855183 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:10.855253 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.855185 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:10.855253 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.855241 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:26.855224588 +0000 UTC m=+34.217663899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:10.855481 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.855260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:26.855251368 +0000 UTC m=+34.217690667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:10.956172 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:10.956137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:10.956376 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.956345 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:15:10.956376 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.956373 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:15:10.956498 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.956386 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:10.956498 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:10.956452 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:26.95643221 +0000 UTC m=+34.318871522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:11.175213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:11.175113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:11.175374 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:11.175264 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:12.175545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:12.175494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:12.175991 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:12.175644 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:12.175991 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:12.175698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:12.175991 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:12.175814 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:13.176310 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:13.176270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:13.176734 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:13.176436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:14.175880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:14.175841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:14.176028 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:14.175844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:14.176028 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:14.175948 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:14.176089 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:14.176028 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:15.175495 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.175167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:15.176043 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:15.175630 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:15.262971 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.262938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjcxr" event={"ID":"6c3e650e-692a-4e51-910a-1772c02cb4bd","Type":"ContainerStarted","Data":"ebac6d3ecb8bb730d1222847e0b04bf2084b032c02d7b5a762e3dc36b6be1d17"} Jul 24 22:15:15.264453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.264428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kp6z4" event={"ID":"b5f35b73-7542-425f-b5b3-1316baa372d8","Type":"ContainerStarted","Data":"82bcb152e27efbf4efde549316374d83c36c49d7ba3f670c70b6fdec27c0c6fd"} Jul 24 22:15:15.265857 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.265831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r85jp" event={"ID":"231e3285-b74a-4a4a-aa5f-0c507a0d9a3a","Type":"ContainerStarted","Data":"9422db9b4b705483498f60abf401b3899a396a0c3dc16350f6664b15a0c34edd"} Jul 24 22:15:15.267493 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.267477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:15:15.267780 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.267764 2576 generic.go:356] "Generic (PLEG): container finished" podID="849c3974-1299-4e99-afd3-1c0691ad5d88" containerID="49dca623a340827d64a48da447b65c04076f598721cf0e5435975d28f4716610" exitCode=1 Jul 24 22:15:15.267836 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.267823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"9503f5ef34b9c4184f3f1fa84d3f5dcef53d16d9c9e13c6251d164ecb8452bae"} Jul 24 22:15:15.267895 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.267874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerDied","Data":"49dca623a340827d64a48da447b65c04076f598721cf0e5435975d28f4716610"} Jul 24 22:15:15.267943 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.267931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"aeda2fa9d45b101f3619a07024f6b361f8c47012e270706cf4e34132515493cc"} Jul 24 22:15:15.269321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.269280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h9tgc" event={"ID":"75481ec4-c026-4496-94fe-050f60955b2f","Type":"ContainerStarted","Data":"8547cd9a5736e560e7ebf5c4f9635d887cae8e870902eda2709d912d1fab4bda"} Jul 24 22:15:15.270750 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.270732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" event={"ID":"977581cc-4ea1-4e86-994f-27963d068e47","Type":"ContainerStarted","Data":"8f210b0d11769d5f23b65079247b13cdd9663ffcc146609a9e34c82d74f06c16"} Jul 24 22:15:15.272185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.272158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6n9w6" event={"ID":"3f7e2ba4-85bc-48de-af19-38e7bcababb4","Type":"ContainerStarted","Data":"3a5b0fce2a76494462d85d5eb013333da39c4aff8f0838c72e2ecca915bef5ba"} Jul 24 22:15:15.273998 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.273974 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="33114e60c9d9923ba75a6ac6bf565f62898a9c04df5021da5e7a003fa78b074c" exitCode=0 Jul 24 22:15:15.274090 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.274010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"33114e60c9d9923ba75a6ac6bf565f62898a9c04df5021da5e7a003fa78b074c"} Jul 24 22:15:15.277840 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.277794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-136.ec2.internal" podStartSLOduration=21.277781242 podStartE2EDuration="21.277781242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:14:58.248398855 +0000 UTC m=+5.610838178" watchObservedRunningTime="2026-07-24 22:15:15.277781242 +0000 UTC m=+22.640220559" Jul 24 22:15:15.278055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.278036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rjcxr" podStartSLOduration=3.716538325 podStartE2EDuration="22.278030584s" totalImagesPullingTime="18.561492259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:15.277271956 +0000 UTC m=+22.639711279" watchObservedRunningTime="2026-07-24 22:15:15.278030584 +0000 UTC m=+22.640469905" Jul 24 22:15:15.295238 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.295201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kp6z4" podStartSLOduration=3.693889859 podStartE2EDuration="22.295187003s" totalImagesPullingTime="18.601297144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:15.294973611 +0000 UTC m=+22.657412923" watchObservedRunningTime="2026-07-24 22:15:15.295187003 +0000 UTC m=+22.657626324" Jul 24 22:15:15.311097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.311057 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h9tgc" podStartSLOduration=3.752451931 podStartE2EDuration="22.311037875s" totalImagesPullingTime="18.558585944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:15.31086299 +0000 UTC m=+22.673302310" watchObservedRunningTime="2026-07-24 22:15:15.311037875 +0000 UTC m=+22.673477195" Jul 24 22:15:15.357594 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.357552 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r85jp" podStartSLOduration=3.755108266 podStartE2EDuration="22.357534634s" totalImagesPullingTime="18.602426368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:15.32939325 +0000 UTC m=+22.691832570" watchObservedRunningTime="2026-07-24 22:15:15.357534634 +0000 UTC m=+22.719973954" Jul 24 22:15:15.379247 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.379205 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6n9w6" podStartSLOduration=8.471287036 podStartE2EDuration="22.37918979s" totalImagesPullingTime="13.907902754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:15.378841633 +0000 UTC m=+22.741280954" watchObservedRunningTime="2026-07-24 22:15:15.37918979 +0000 UTC m=+22.741629110" Jul 24 22:15:15.451072 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.451032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:15.525401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:15.525367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.175228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.175188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:16.175228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.175204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:16.175456 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:16.175317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:16.175456 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:16.175433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:16.241192 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.241076 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 22:15:16.254784 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.254761 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:15:16.255346 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.255330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:15:16.277387 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.277308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.279229 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.279207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:15:16.279695 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.279668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"263164ffa014dbdf1d189cf414cad114a606b7fcb7f7918413ee2a0a4964c47d"} Jul 24 22:15:16.279793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.279740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"3477ca9740dc0f36020fe67a3611b3b40acc7086e0038c797bd980aecf644176"} Jul 24 22:15:16.279793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.279774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"893dee6b4a3fc744c511684749f49870a6641f409f4b837e15d47a999695c0d7"} Jul 24 22:15:16.281712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.281278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fnbln" event={"ID":"48bbbb33-5bbe-4006-82af-7007a76c42a8","Type":"ContainerStarted","Data":"d546f0d6d5b7452dbde4191e207954f02ba1b4372b7d94a6cb0f512dec7c89b8"} Jul 24 22:15:16.283172 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.283127 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" event={"ID":"977581cc-4ea1-4e86-994f-27963d068e47","Type":"ContainerStarted","Data":"20da1e4ef41f851e7cde924f57be1de99ae4f3b0da14afc0825865d06fad03a5"} Jul 24 22:15:16.304744 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.304688 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fnbln" podStartSLOduration=4.7080320570000005 podStartE2EDuration="23.304667766s" totalImagesPullingTime="18.596635709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:16.304268407 +0000 UTC m=+23.666707730" watchObservedRunningTime="2026-07-24 22:15:16.304667766 +0000 UTC m=+23.667107088" Jul 24 22:15:16.350131 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.350095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.427207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.427168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.487226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.487192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.792864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.792822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:16.891441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:16.891405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:17.076032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.075865 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T22:15:16.241177842Z","UUID":"d98ff311-a1d3-4920-9d2b-a73a3923c3a1","Handler":null,"Name":"","Endpoint":""} Jul 24 22:15:17.077539 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.077514 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 22:15:17.077681 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.077559 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 22:15:17.176106 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.176068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:17.176286 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:17.176239 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:17.261618 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.261578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:15:17.262225 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.262205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6n9w6" Jul 24 22:15:17.338317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:17.338265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:18.175037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.174849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:18.175199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.175173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:18.175334 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:18.175287 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:18.175473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.175340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:18.175473 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:18.175451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:18.275110 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.275071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:18.290719 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.290694 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:15:18.291179 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.291147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"04d04b359f94533a68988c84420887b16bcf51fb4b725276f2a0428d33abe842"} Jul 24 22:15:18.293874 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:18.293850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" event={"ID":"977581cc-4ea1-4e86-994f-27963d068e47","Type":"ContainerStarted","Data":"99db9f57fb381b3ad2f51327abfe216253b5aead9c3fe57e737ef98039723b59"} Jul 24 22:15:19.175779 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:19.175745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:19.175998 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:19.175896 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:20.175977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.175725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:20.175977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.175773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:20.175977 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:20.175969 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:20.176710 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:20.176055 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:20.301371 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.301344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:15:20.301727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.301701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"0391fa34bddbc670264cf0502953974a1e11408d034ca49637ff8b442f043a01"} Jul 24 22:15:20.301954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.301929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:20.302200 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.302183 2576 scope.go:117] "RemoveContainer" containerID="49dca623a340827d64a48da447b65c04076f598721cf0e5435975d28f4716610" Jul 24 22:15:20.303760 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.303736 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="ccf5e7b84fb0bba77e4407c5df936329012fbe46bc7f712018834bbd7cd10c5c" exitCode=0 Jul 24 22:15:20.303858 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.303773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"ccf5e7b84fb0bba77e4407c5df936329012fbe46bc7f712018834bbd7cd10c5c"} Jul 24 22:15:20.320590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.320566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:20.330222 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:20.330184 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99p7d" podStartSLOduration=6.635341468 podStartE2EDuration="27.330169486s" totalImagesPullingTime="20.694828018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:18.328877281 +0000 UTC m=+25.691316617" watchObservedRunningTime="2026-07-24 22:15:20.330169486 +0000 UTC m=+27.692608856" Jul 24 22:15:21.175923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.175835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:21.176073 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:21.175959 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:21.312778 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.312757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:15:21.313169 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.313139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" event={"ID":"849c3974-1299-4e99-afd3-1c0691ad5d88","Type":"ContainerStarted","Data":"d5a39f7594b96b27bf5e78b1eb780e21a17f9639314076d9b0236910dc9c208d"} Jul 24 22:15:21.313436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.313400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:21.313590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.313572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:21.316552 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.316531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerStarted","Data":"7ecd0d61ba88620643602be814c2a1cc2a250fdbc8f895e11d4ea12865109d1a"} Jul 24 22:15:21.334762 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.334739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:21.394227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.394177 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" podStartSLOduration=9.734591229 podStartE2EDuration="28.394161817s" totalImagesPullingTime="18.659570588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:21.361474245 +0000 UTC m=+28.723913568" watchObservedRunningTime="2026-07-24 22:15:21.394161817 +0000 UTC m=+28.756601162" Jul 24 22:15:21.833318 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.833099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t42k6"] Jul 24 22:15:21.833475 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.833439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:21.833582 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:21.833562 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:21.837464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.837438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqshq"] Jul 24 22:15:21.837605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.837557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:21.837677 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:21.837654 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:21.842035 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.842011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-84ckx"] Jul 24 22:15:21.842140 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:21.842114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:21.842225 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:21.842208 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:22.320708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:22.320673 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="7ecd0d61ba88620643602be814c2a1cc2a250fdbc8f895e11d4ea12865109d1a" exitCode=0 Jul 24 22:15:22.321163 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:22.320749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"7ecd0d61ba88620643602be814c2a1cc2a250fdbc8f895e11d4ea12865109d1a"} Jul 24 22:15:23.176659 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:23.176623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:23.176802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:23.176694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:23.176802 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:23.176711 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:23.176911 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:23.176820 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:23.325534 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:23.325443 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="c790e8354609a9cb3ffd2225716b059a76709d0df5e8a1446f2975b7aa1afbfe" exitCode=0 Jul 24 22:15:23.325534 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:23.325519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"c790e8354609a9cb3ffd2225716b059a76709d0df5e8a1446f2975b7aa1afbfe"} Jul 24 22:15:24.175693 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:24.175614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:24.175848 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:24.175737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:25.175662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:25.175626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:25.176385 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:25.175681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:25.176385 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:25.175772 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:25.176385 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:25.175863 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:25.496565 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:25.496526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:25.613608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:25.613559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:26.175829 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.175801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:26.176380 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.175960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqshq" podUID="649a5a5f-f958-49f3-929b-dfe6460e3002" Jul 24 22:15:26.401852 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.401813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:26.462586 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.462528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:26.525430 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.525400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:26.873043 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.872944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:26.873043 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.872996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:26.873241 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.873125 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:26.873241 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.873126 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:26.873241 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.873174 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret podName:0980d09d-775a-4f2a-82d9-7a5167813f50 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:58.87315972 +0000 UTC m=+66.235599019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret") pod "global-pull-secret-syncer-84ckx" (UID: "0980d09d-775a-4f2a-82d9-7a5167813f50") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:26.873241 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.873188 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:58.873181104 +0000 UTC m=+66.235620403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:26.973606 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:26.973570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:26.973799 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.973701 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:15:26.973799 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.973720 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:15:26.973799 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.973730 2576 projected.go:196] Error preparing data for projected volume kube-api-access-87mjk for pod openshift-network-diagnostics/network-check-target-gqshq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:26.973799 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:26.973785 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk podName:649a5a5f-f958-49f3-929b-dfe6460e3002 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:58.973766863 +0000 UTC m=+66.336206181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87mjk" (UniqueName: "kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk") pod "network-check-target-gqshq" (UID: "649a5a5f-f958-49f3-929b-dfe6460e3002") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:27.044880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.044843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:27.149784 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.149705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:27.179359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.179331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:27.179779 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:27.179443 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-84ckx" podUID="0980d09d-775a-4f2a-82d9-7a5167813f50" Jul 24 22:15:27.179779 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.179499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:27.179779 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:27.179630 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t42k6" podUID="a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b" Jul 24 22:15:27.429939 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.429864 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-136.ec2.internal" event="NodeReady" Jul 24 22:15:27.430139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.430001 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 22:15:27.450148 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.450114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:27.483158 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.483127 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpljc"] Jul 24 22:15:27.501453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.501427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lds7x"] Jul 24 22:15:27.501635 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.501558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.505123 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.505102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 22:15:27.505237 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.505176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wnkgk\"" Jul 24 22:15:27.505817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.505795 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 22:15:27.516690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.516669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpljc"] Jul 24 22:15:27.516813 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.516712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lds7x"] Jul 24 22:15:27.516868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.516807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.519536 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.519516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 22:15:27.519620 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.519565 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ck9v8\"" Jul 24 22:15:27.519908 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.519871 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 22:15:27.520087 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.520044 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 22:15:27.533967 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.533944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:27.577928 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.577900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.578045 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.577955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386afff3-1e58-4e1b-95be-2cc4087b78f7-config-volume\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.578045 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.578021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcq7s\" (UniqueName: \"kubernetes.io/projected/386afff3-1e58-4e1b-95be-2cc4087b78f7-kube-api-access-bcq7s\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.578117 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.578056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386afff3-1e58-4e1b-95be-2cc4087b78f7-tmp-dir\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.615480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.615445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:27.679370 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcq7s\" (UniqueName: \"kubernetes.io/projected/386afff3-1e58-4e1b-95be-2cc4087b78f7-kube-api-access-bcq7s\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.679370 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386afff3-1e58-4e1b-95be-2cc4087b78f7-tmp-dir\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.679550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.679550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf12a677-1a57-4dbf-803f-90f6bdd3b722-cert\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.679550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzw7k\" (UniqueName: \"kubernetes.io/projected/cf12a677-1a57-4dbf-803f-90f6bdd3b722-kube-api-access-dzw7k\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.679550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386afff3-1e58-4e1b-95be-2cc4087b78f7-config-volume\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.679758 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:27.679603 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:27.679758 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:27.679677 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:28.179655352 +0000 UTC m=+35.542094858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:27.679758 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386afff3-1e58-4e1b-95be-2cc4087b78f7-tmp-dir\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.680013 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.679974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386afff3-1e58-4e1b-95be-2cc4087b78f7-config-volume\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.691662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.691637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcq7s\" (UniqueName: \"kubernetes.io/projected/386afff3-1e58-4e1b-95be-2cc4087b78f7-kube-api-access-bcq7s\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:27.780213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.780182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf12a677-1a57-4dbf-803f-90f6bdd3b722-cert\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.780213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.780215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw7k\" (UniqueName: \"kubernetes.io/projected/cf12a677-1a57-4dbf-803f-90f6bdd3b722-kube-api-access-dzw7k\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.782734 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.782711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf12a677-1a57-4dbf-803f-90f6bdd3b722-cert\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.789604 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.789582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw7k\" (UniqueName: \"kubernetes.io/projected/cf12a677-1a57-4dbf-803f-90f6bdd3b722-kube-api-access-dzw7k\") pod \"ingress-canary-lds7x\" (UID: \"cf12a677-1a57-4dbf-803f-90f6bdd3b722\") " pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:27.827655 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:27.827623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lds7x" Jul 24 22:15:28.083565 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.083348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lds7x"] Jul 24 22:15:28.088166 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:15:28.088145 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf12a677_1a57_4dbf_803f_90f6bdd3b722.slice/crio-19ce488ad3e4323653dbea61a713cb3850eb15d04e2511d660a72d58bac735dd WatchSource:0}: Error finding container 19ce488ad3e4323653dbea61a713cb3850eb15d04e2511d660a72d58bac735dd: Status 404 returned error can't find the container with id 19ce488ad3e4323653dbea61a713cb3850eb15d04e2511d660a72d58bac735dd Jul 24 22:15:28.175079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.175055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:28.177767 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.177741 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:15:28.177880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.177741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rtdx2\"" Jul 24 22:15:28.177880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.177765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:15:28.182029 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.182006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:28.182369 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:28.182121 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:28.182369 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:28.182166 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:29.182153584 +0000 UTC m=+36.544592883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:28.343414 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.343384 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="2568572d402ecf376357ea3cac5d19a14a8d56f2a737861aaa881e5e8292e4c7" exitCode=0 Jul 24 22:15:28.343582 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.343465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"2568572d402ecf376357ea3cac5d19a14a8d56f2a737861aaa881e5e8292e4c7"} Jul 24 22:15:28.344824 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:28.344583 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lds7x" event={"ID":"cf12a677-1a57-4dbf-803f-90f6bdd3b722","Type":"ContainerStarted","Data":"19ce488ad3e4323653dbea61a713cb3850eb15d04e2511d660a72d58bac735dd"} Jul 24 22:15:29.175281 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.175247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:29.175575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.175311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:29.178449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.177986 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:15:29.178449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.178063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:15:29.178449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.178167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-767d7\"" Jul 24 22:15:29.190266 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.190091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:29.190266 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:29.190190 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:29.190266 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:29.190254 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:31.190239317 +0000 UTC m=+38.552678617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:29.350150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.350116 2576 generic.go:356] "Generic (PLEG): container finished" podID="afc8daf1-aced-453d-99be-31e6ca0a148d" containerID="b0202d5550abf1230022b2ccbfda7fbf7754eb8749ab393430e34c2c5d7b76e4" exitCode=0 Jul 24 22:15:29.350312 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:29.350173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerDied","Data":"b0202d5550abf1230022b2ccbfda7fbf7754eb8749ab393430e34c2c5d7b76e4"} Jul 24 22:15:30.356469 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:30.356238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" event={"ID":"afc8daf1-aced-453d-99be-31e6ca0a148d","Type":"ContainerStarted","Data":"e51f6d06cfbc772a97485ae112c6b86841d4a5b1cff1e18edbd0fd7a568bf612"} Jul 24 22:15:30.384348 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:30.384280 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8rkhl" podStartSLOduration=8.378952353 podStartE2EDuration="37.38426312s" totalImagesPullingTime="29.005310767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:30.382615314 +0000 UTC m=+37.745054634" watchObservedRunningTime="2026-07-24 22:15:30.38426312 +0000 UTC m=+37.746702443" Jul 24 22:15:30.504573 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:30.504542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:30.605053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:30.605029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:31.206723 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:31.206689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:31.206907 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:31.206811 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:31.206907 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:31.206868 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:35.206852256 +0000 UTC m=+42.569291562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:31.359568 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:31.359535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lds7x" event={"ID":"cf12a677-1a57-4dbf-803f-90f6bdd3b722","Type":"ContainerStarted","Data":"56cf3f76deab38df242f42c9e580d282958ceaea09c2b6d3523c6ac637561e4c"} Jul 24 22:15:31.377468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:31.377426 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lds7x" podStartSLOduration=1.883170595 podStartE2EDuration="4.377410868s" totalImagesPullingTime="2.494240273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:31.377105333 +0000 UTC m=+38.739544653" watchObservedRunningTime="2026-07-24 22:15:31.377410868 +0000 UTC m=+38.739850188" Jul 24 22:15:33.264158 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:33.264128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:33.328904 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:33.328875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:35.235464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:35.235422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:35.235817 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:35.235538 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:35.235817 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:35.235590 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:43.235577241 +0000 UTC m=+50.598016540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:38.100775 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:38.100736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:38.200790 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:38.200757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:39.276315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:39.276272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:39.355757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:39.355726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:42.542679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:42.542643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:42.642452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:42.642418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:43.026971 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:43.026935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:43.088869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:43.088827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:43.293164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:43.293073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:43.293347 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:43.293224 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:43.293347 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:43.293316 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:59.293281574 +0000 UTC m=+66.655720872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:45.565084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.565048 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654"] Jul 24 22:15:45.587388 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.587361 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp"] Jul 24 22:15:45.587551 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.587477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.590024 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.590003 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:15:45.590185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.590025 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 22:15:45.590185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.590051 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 22:15:45.590185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.590068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 22:15:45.590185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.590027 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4s79j\"" Jul 24 22:15:45.608009 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.607990 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-sdhsp"] Jul 24 22:15:45.608116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.608099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.610545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.610524 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:15:45.610650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.610548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r5k7c\"" Jul 24 22:15:45.610650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.610549 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 22:15:45.610650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.610554 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 22:15:45.628052 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.628031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654"] Jul 24 22:15:45.628149 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.628061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp"] Jul 24 22:15:45.628149 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.628084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-sdhsp"] Jul 24 22:15:45.628149 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.628146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.630765 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.630747 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:15:45.630871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.630847 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7f9rm\"" Jul 24 22:15:45.630939 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.630892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 22:15:45.631007 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.630992 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 22:15:45.631174 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.631162 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 22:15:45.635933 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.635918 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 22:15:45.711113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.711113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33138bc3-3754-4211-85fc-ed955daaaa5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.711342 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-trusted-ca\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.711342 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33138bc3-3754-4211-85fc-ed955daaaa5f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.711342 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv9c\" (UniqueName: \"kubernetes.io/projected/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-kube-api-access-bfv9c\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.711448 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-config\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.711448 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-serving-cert\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.711448 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmn6\" (UniqueName: \"kubernetes.io/projected/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-kube-api-access-pqmn6\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.711448 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.711424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2df\" (UniqueName: \"kubernetes.io/projected/33138bc3-3754-4211-85fc-ed955daaaa5f-kube-api-access-7p2df\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.743836 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.743806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:45.772136 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.772108 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:15:45.793952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.793923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:15:45.794078 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.794020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.796542 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.796518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 22:15:45.796673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.796664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 22:15:45.796744 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.796722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 22:15:45.797011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.796996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sr77c\"" Jul 24 22:15:45.802065 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.802047 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 22:15:45.812236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33138bc3-3754-4211-85fc-ed955daaaa5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.812321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-trusted-ca\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.812321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33138bc3-3754-4211-85fc-ed955daaaa5f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.812321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv9c\" (UniqueName: \"kubernetes.io/projected/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-kube-api-access-bfv9c\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.812481 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-config\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.812481 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-serving-cert\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.812481 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmn6\" (UniqueName: \"kubernetes.io/projected/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-kube-api-access-pqmn6\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.812481 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2df\" (UniqueName: \"kubernetes.io/projected/33138bc3-3754-4211-85fc-ed955daaaa5f-kube-api-access-7p2df\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.812665 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.812665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:45.812631 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:15:45.812765 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:45.812727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls podName:80ac1f52-2f4f-46e7-b6e8-2d6cce205368 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:46.312711294 +0000 UTC m=+53.675150609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls") pod "cluster-samples-operator-6748489656-6kcmp" (UID: "80ac1f52-2f4f-46e7-b6e8-2d6cce205368") : secret "samples-operator-tls" not found Jul 24 22:15:45.812939 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.812903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33138bc3-3754-4211-85fc-ed955daaaa5f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.813190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.813164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-config\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.813272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.813231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-trusted-ca\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.816204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.816151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33138bc3-3754-4211-85fc-ed955daaaa5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.816204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.816178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-serving-cert\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.824582 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.824559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:45.825179 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.825158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv9c\" (UniqueName: \"kubernetes.io/projected/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-kube-api-access-bfv9c\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:45.825322 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.825281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmn6\" (UniqueName: \"kubernetes.io/projected/8a58ca8d-ae71-448c-8fd7-0a0f1ad98240-kube-api-access-pqmn6\") pod \"console-operator-584dcb6b54-sdhsp\" (UID: \"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240\") " pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:45.825651 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.825629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2df\" (UniqueName: \"kubernetes.io/projected/33138bc3-3754-4211-85fc-ed955daaaa5f-kube-api-access-7p2df\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt654\" (UID: \"33138bc3-3754-4211-85fc-ed955daaaa5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.896706 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.896670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" Jul 24 22:15:45.913007 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.912974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913007 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ls2\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913328 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913328 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.913328 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.913239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:45.935956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:45.935920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:46.014519 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014519 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ls2\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.014865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.015698 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.014985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.015698 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.015214 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:46.015698 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.015236 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5f867788-dtsds: secret "image-registry-tls" not found Jul 24 22:15:46.015698 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.015316 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls podName:617fe8fd-7289-4d0b-8384-17094658b85b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:46.515273807 +0000 UTC m=+53.877713108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls") pod "image-registry-6b5f867788-dtsds" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b") : secret "image-registry-tls" not found Jul 24 22:15:46.015698 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.015329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.016599 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.016576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.018540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.018506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.018540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.018506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.027596 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.027570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ls2\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.031094 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.031069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.075262 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.075183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654"] Jul 24 22:15:46.078709 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:15:46.078679 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33138bc3_3754_4211_85fc_ed955daaaa5f.slice/crio-3bae4ff75741d1c8b56e021c5ff421cbafe248c8610e3193b37766e439f2d3a7 WatchSource:0}: Error finding container 3bae4ff75741d1c8b56e021c5ff421cbafe248c8610e3193b37766e439f2d3a7: Status 404 returned error can't find the container with id 3bae4ff75741d1c8b56e021c5ff421cbafe248c8610e3193b37766e439f2d3a7 Jul 24 22:15:46.081178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.081160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-sdhsp"] Jul 24 22:15:46.084227 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:15:46.084209 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a58ca8d_ae71_448c_8fd7_0a0f1ad98240.slice/crio-d28692bd1429acaa05097e574c2fcdd5f694d59ac567d60b2b572299a98e2a73 WatchSource:0}: Error finding container d28692bd1429acaa05097e574c2fcdd5f694d59ac567d60b2b572299a98e2a73: Status 404 returned error can't find the container with id d28692bd1429acaa05097e574c2fcdd5f694d59ac567d60b2b572299a98e2a73 Jul 24 22:15:46.316336 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.316271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:46.316493 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.316404 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:15:46.316493 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.316468 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls podName:80ac1f52-2f4f-46e7-b6e8-2d6cce205368 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:47.316452788 +0000 UTC m=+54.678892087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls") pod "cluster-samples-operator-6748489656-6kcmp" (UID: "80ac1f52-2f4f-46e7-b6e8-2d6cce205368") : secret "samples-operator-tls" not found Jul 24 22:15:46.392560 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.392475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" event={"ID":"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240","Type":"ContainerStarted","Data":"d28692bd1429acaa05097e574c2fcdd5f694d59ac567d60b2b572299a98e2a73"} Jul 24 22:15:46.393530 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.393507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" event={"ID":"33138bc3-3754-4211-85fc-ed955daaaa5f","Type":"ContainerStarted","Data":"3bae4ff75741d1c8b56e021c5ff421cbafe248c8610e3193b37766e439f2d3a7"} Jul 24 22:15:46.518362 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:46.518315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:46.518527 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.518490 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:46.518527 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.518514 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5f867788-dtsds: secret "image-registry-tls" not found Jul 24 22:15:46.518593 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:46.518574 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls podName:617fe8fd-7289-4d0b-8384-17094658b85b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:47.518556007 +0000 UTC m=+54.880995326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls") pod "image-registry-6b5f867788-dtsds" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b") : secret "image-registry-tls" not found Jul 24 22:15:47.326709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:47.326483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:47.327164 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:47.326721 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:15:47.327164 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:47.326816 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls podName:80ac1f52-2f4f-46e7-b6e8-2d6cce205368 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.326793747 +0000 UTC m=+56.689233048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls") pod "cluster-samples-operator-6748489656-6kcmp" (UID: "80ac1f52-2f4f-46e7-b6e8-2d6cce205368") : secret "samples-operator-tls" not found Jul 24 22:15:47.529042 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:47.529005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:47.529226 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:47.529197 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:47.529226 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:47.529219 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5f867788-dtsds: secret "image-registry-tls" not found Jul 24 22:15:47.529420 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:47.529312 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls podName:617fe8fd-7289-4d0b-8384-17094658b85b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.529271395 +0000 UTC m=+56.891710707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls") pod "image-registry-6b5f867788-dtsds" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b") : secret "image-registry-tls" not found Jul 24 22:15:49.071069 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.071030 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8d7hv"] Jul 24 22:15:49.073274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.073250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.076695 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.076673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 22:15:49.077031 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.077013 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 22:15:49.077428 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.077414 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 22:15:49.077989 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.077974 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nftg\"" Jul 24 22:15:49.077989 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.077985 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 22:15:49.115731 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.115695 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8d7hv"] Jul 24 22:15:49.156892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.156862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:49.217669 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.217628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:49.242723 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.242688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4bb5c83-712e-430d-aea1-882e82e2294d-crio-socket\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.242843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.242738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4bb5c83-712e-430d-aea1-882e82e2294d-data-volume\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.242843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.242802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.243013 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.242863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.243013 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.242907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bvp\" (UniqueName: \"kubernetes.io/projected/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-api-access-v4bvp\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bvp\" (UniqueName: \"kubernetes.io/projected/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-api-access-v4bvp\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4bb5c83-712e-430d-aea1-882e82e2294d-crio-socket\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4bb5c83-712e-430d-aea1-882e82e2294d-data-volume\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4bb5c83-712e-430d-aea1-882e82e2294d-crio-socket\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344540 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344563 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344598 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.844582123 +0000 UTC m=+57.207021549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344612 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls podName:80ac1f52-2f4f-46e7-b6e8-2d6cce205368 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:53.344605808 +0000 UTC m=+60.707045107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls") pod "cluster-samples-operator-6748489656-6kcmp" (UID: "80ac1f52-2f4f-46e7-b6e8-2d6cce205368") : secret "samples-operator-tls" not found Jul 24 22:15:49.344665 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344634 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:49.344944 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.344720 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.844707259 +0000 UTC m=+57.207146574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : configmap "kube-rbac-proxy" not found Jul 24 22:15:49.344944 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.344725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4bb5c83-712e-430d-aea1-882e82e2294d-data-volume\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.353084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.353064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bvp\" (UniqueName: \"kubernetes.io/projected/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-api-access-v4bvp\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.402260 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.402231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/0.log" Jul 24 22:15:49.402423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.402271 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" containerID="3edce94eea7d2150a09aa118c88fa8b319d9a881ff3cbae1ab6f9c6f57af734a" exitCode=255 Jul 24 22:15:49.402423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.402361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" event={"ID":"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240","Type":"ContainerDied","Data":"3edce94eea7d2150a09aa118c88fa8b319d9a881ff3cbae1ab6f9c6f57af734a"} Jul 24 22:15:49.402646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.402626 2576 scope.go:117] "RemoveContainer" containerID="3edce94eea7d2150a09aa118c88fa8b319d9a881ff3cbae1ab6f9c6f57af734a" Jul 24 22:15:49.403877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.403851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" event={"ID":"33138bc3-3754-4211-85fc-ed955daaaa5f","Type":"ContainerStarted","Data":"cefb3bef7a671727daf25b315cec23a118b3f62ceb868b2bcc8447b35ec0a2ab"} Jul 24 22:15:49.434892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.434852 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" podStartSLOduration=2.057035801 podStartE2EDuration="4.434836667s" totalImagesPullingTime="2.377800866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:49.434368026 +0000 UTC m=+56.796807351" watchObservedRunningTime="2026-07-24 22:15:49.434836667 +0000 UTC m=+56.797275966" Jul 24 22:15:49.546878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.546462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:49.546878 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.546777 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:49.546878 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.546798 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5f867788-dtsds: secret "image-registry-tls" not found Jul 24 22:15:49.546878 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.546874 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls podName:617fe8fd-7289-4d0b-8384-17094658b85b nodeName:}" failed. No retries permitted until 2026-07-24 22:15:53.546853465 +0000 UTC m=+60.909292789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls") pod "image-registry-6b5f867788-dtsds" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b") : secret "image-registry-tls" not found Jul 24 22:15:49.837968 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.837935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:49.850102 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.850066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.850239 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.850116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:49.850239 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.850215 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.850333 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.850247 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:49.850333 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.850269 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:50.850254758 +0000 UTC m=+58.212694061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.850403 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:49.850338 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:50.850327421 +0000 UTC m=+58.212766720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : configmap "kube-rbac-proxy" not found Jul 24 22:15:49.936795 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:49.936762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:50.002991 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.002945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:50.407555 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.407530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/1.log" Jul 24 22:15:50.407940 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.407887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/0.log" Jul 24 22:15:50.407940 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.407927 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" exitCode=255 Jul 24 22:15:50.408055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.408019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" event={"ID":"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240","Type":"ContainerDied","Data":"8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54"} Jul 24 22:15:50.408103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.408076 2576 scope.go:117] "RemoveContainer" containerID="3edce94eea7d2150a09aa118c88fa8b319d9a881ff3cbae1ab6f9c6f57af734a" Jul 24 22:15:50.408368 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.408347 2576 scope.go:117] "RemoveContainer" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" Jul 24 22:15:50.408544 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:50.408527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-sdhsp_openshift-console-operator(8a58ca8d-ae71-448c-8fd7-0a0f1ad98240)\"" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podUID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" Jul 24 22:15:50.858671 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.858627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:50.858870 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:50.858697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:50.858870 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:50.858785 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:50.858870 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:50.858824 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:50.858870 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:50.858855 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:52.85883969 +0000 UTC m=+60.221278988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:50.859023 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:50.858874 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:52.858861355 +0000 UTC m=+60.221300653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : configmap "kube-rbac-proxy" not found Jul 24 22:15:51.250984 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:51.250960 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjcxr_6c3e650e-692a-4e51-910a-1772c02cb4bd/dns-node-resolver/0.log" Jul 24 22:15:51.400889 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:51.400855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:51.412054 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:51.412030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/1.log" Jul 24 22:15:51.412456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:51.412389 2576 scope.go:117] "RemoveContainer" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" Jul 24 22:15:51.412547 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:51.412527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-sdhsp_openshift-console-operator(8a58ca8d-ae71-448c-8fd7-0a0f1ad98240)\"" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podUID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" Jul 24 22:15:52.251011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.250980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h9tgc_75481ec4-c026-4496-94fe-050f60955b2f/node-ca/0.log" Jul 24 22:15:52.445212 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.445177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:52.501206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.501117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:52.562808 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.562776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:52.874993 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.874894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:52.874993 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.874958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:52.875198 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:52.875029 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:52.875198 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:52.875057 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:52.875198 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:52.875108 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:56.875086656 +0000 UTC m=+64.237525956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : configmap "kube-rbac-proxy" not found Jul 24 22:15:52.875198 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:52.875123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:56.875116816 +0000 UTC m=+64.237556115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:52.881401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.881378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:52.981528 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:52.981496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:53.052192 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.052166 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lds7x_cf12a677-1a57-4dbf-803f-90f6bdd3b722/serve-healthcheck-canary/0.log" Jul 24 22:15:53.147236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.147146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:53.250155 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.250123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:53.337072 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.337048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ffrc" Jul 24 22:15:53.377653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.377615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:15:53.377841 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:53.377801 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:15:53.377915 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:53.377866 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls podName:80ac1f52-2f4f-46e7-b6e8-2d6cce205368 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:01.377847663 +0000 UTC m=+68.740286963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls") pod "cluster-samples-operator-6748489656-6kcmp" (UID: "80ac1f52-2f4f-46e7-b6e8-2d6cce205368") : secret "samples-operator-tls" not found Jul 24 22:15:53.512989 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.512955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:53.579125 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.579094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:15:53.579273 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:53.579245 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:53.579273 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:53.579264 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5f867788-dtsds: secret "image-registry-tls" not found Jul 24 22:15:53.579398 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:53.579342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls podName:617fe8fd-7289-4d0b-8384-17094658b85b nodeName:}" failed. No retries permitted until 2026-07-24 22:16:01.579325908 +0000 UTC m=+68.941765206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls") pod "image-registry-6b5f867788-dtsds" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b") : secret "image-registry-tls" not found Jul 24 22:15:53.612803 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:53.612771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:54.784157 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:54.784059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:54.844933 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:54.844903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:55.782490 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:55.782460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:55.897619 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:55.897589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:55.936561 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:55.936521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:55.936673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:55.936589 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:15:55.936926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:55.936914 2576 scope.go:117] "RemoveContainer" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" Jul 24 22:15:55.937067 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:55.937050 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-sdhsp_openshift-console-operator(8a58ca8d-ae71-448c-8fd7-0a0f1ad98240)\"" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podUID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" Jul 24 22:15:56.004054 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.004023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:56.466853 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.466821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:56.566123 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.566089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:56.713017 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.712982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:56.811335 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.811250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:56.905058 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.905016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:56.905515 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:56.905140 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:56.905515 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.905150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:15:56.905515 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:56.905235 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:16:04.905211737 +0000 UTC m=+72.267651054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : configmap "kube-rbac-proxy" not found Jul 24 22:15:56.905515 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:56.905321 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:56.905515 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:56.905398 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:16:04.905384907 +0000 UTC m=+72.267824205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:56.911843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:56.911801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:57.181429 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:57.181344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:58.189513 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.189477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:58.564189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.564158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:58.764687 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.764655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:58.920945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.920856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:15:58.920945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.920930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:58.923851 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.923831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:15:58.923973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.923838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:15:58.931951 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:58.931924 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 22:15:58.932044 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:58.931993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs podName:a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b nodeName:}" failed. No retries permitted until 2026-07-24 22:17:02.931976103 +0000 UTC m=+130.294415401 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs") pod "network-metrics-daemon-t42k6" (UID: "a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b") : secret "metrics-daemon-secret" not found Jul 24 22:15:58.934203 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.934181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0980d09d-775a-4f2a-82d9-7a5167813f50-original-pull-secret\") pod \"global-pull-secret-syncer-84ckx\" (UID: \"0980d09d-775a-4f2a-82d9-7a5167813f50\") " pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:58.964717 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:58.964685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:59.021954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.021917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:59.024748 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.024729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:15:59.035145 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.035122 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:15:59.045114 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.045089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mjk\" (UniqueName: \"kubernetes.io/projected/649a5a5f-f958-49f3-929b-dfe6460e3002-kube-api-access-87mjk\") pod \"network-check-target-gqshq\" (UID: \"649a5a5f-f958-49f3-929b-dfe6460e3002\") " pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:59.138560 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.138527 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rtdx2\"" Jul 24 22:15:59.146401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.146381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:15:59.193441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.193407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-84ckx" Jul 24 22:15:59.223763 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.223727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:59.305744 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.305717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqshq"] Jul 24 22:15:59.321457 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:15:59.321432 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649a5a5f_f958_49f3_929b_dfe6460e3002.slice/crio-8a3aa713d0abf4587971b263a70a9686c61ccef79a65d3beb990912f7101d61c WatchSource:0}: Error finding container 8a3aa713d0abf4587971b263a70a9686c61ccef79a65d3beb990912f7101d61c: Status 404 returned error can't find the container with id 8a3aa713d0abf4587971b263a70a9686c61ccef79a65d3beb990912f7101d61c Jul 24 22:15:59.323789 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.323685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:15:59.323848 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:59.323821 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:59.323894 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:15:59.323878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls podName:386afff3-1e58-4e1b-95be-2cc4087b78f7 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:31.323862186 +0000 UTC m=+98.686301485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls") pod "dns-default-vpljc" (UID: "386afff3-1e58-4e1b-95be-2cc4087b78f7") : secret "dns-default-metrics-tls" not found Jul 24 22:15:59.352913 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.352884 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-84ckx"] Jul 24 22:15:59.356108 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:15:59.356080 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0980d09d_775a_4f2a_82d9_7a5167813f50.slice/crio-4ffccd8bbe1deff80eb6f451e72366cb9728c5e94d30fccf70ec51168f6981a8 WatchSource:0}: Error finding container 4ffccd8bbe1deff80eb6f451e72366cb9728c5e94d30fccf70ec51168f6981a8: Status 404 returned error can't find the container with id 4ffccd8bbe1deff80eb6f451e72366cb9728c5e94d30fccf70ec51168f6981a8 Jul 24 22:15:59.365956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.365929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:59.432330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.432235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqshq" event={"ID":"649a5a5f-f958-49f3-929b-dfe6460e3002","Type":"ContainerStarted","Data":"8a3aa713d0abf4587971b263a70a9686c61ccef79a65d3beb990912f7101d61c"} Jul 24 22:15:59.433406 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.433386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-84ckx" event={"ID":"0980d09d-775a-4f2a-82d9-7a5167813f50","Type":"ContainerStarted","Data":"4ffccd8bbe1deff80eb6f451e72366cb9728c5e94d30fccf70ec51168f6981a8"} Jul 24 22:15:59.970423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:15:59.970383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:00.161216 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:00.161178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:00.288970 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:00.288880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:00.568135 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:00.568049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:00.762284 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:00.762204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:00.961273 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:00.961243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:01.060110 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.060074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:01.359520 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.359442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:01.443850 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.443820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:16:01.446930 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.446907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/80ac1f52-2f4f-46e7-b6e8-2d6cce205368-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-6kcmp\" (UID: \"80ac1f52-2f4f-46e7-b6e8-2d6cce205368\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:16:01.518634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.518599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r5k7c\"" Jul 24 22:16:01.526866 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.526835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" Jul 24 22:16:01.645794 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.645712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:01.648712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.648679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"image-registry-6b5f867788-dtsds\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:01.706428 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.706393 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sr77c\"" Jul 24 22:16:01.714050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.714011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:01.968450 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:01.968420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:02.262442 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:02.262355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:02.370048 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:02.370018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:02.565990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:02.565905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:02.774963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:02.774929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:02.979310 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:02.979258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:03.578879 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:03.578847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:03.760528 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:03.760490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:03.913411 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:03.913382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp"] Jul 24 22:16:03.928830 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:03.928769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:16:03.936362 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:03.936330 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617fe8fd_7289_4d0b_8384_17094658b85b.slice/crio-05e3229dae0eee3784114e5b959417c62b7cad18b83407d6f61ce418f678e36e WatchSource:0}: Error finding container 05e3229dae0eee3784114e5b959417c62b7cad18b83407d6f61ce418f678e36e: Status 404 returned error can't find the container with id 05e3229dae0eee3784114e5b959417c62b7cad18b83407d6f61ce418f678e36e Jul 24 22:16:04.154334 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.154200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:04.450121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.450087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" event={"ID":"80ac1f52-2f4f-46e7-b6e8-2d6cce205368","Type":"ContainerStarted","Data":"33cc57faedc4503a9dc3977854c0a86e11e6a12a7319a4e913ae9f637893c717"} Jul 24 22:16:04.451617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.451588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqshq" event={"ID":"649a5a5f-f958-49f3-929b-dfe6460e3002","Type":"ContainerStarted","Data":"bd62e4014b2e650fb1aa3c95f4e92b69b2f70b46afda70beb03c0dbe4509489a"} Jul 24 22:16:04.451731 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.451664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:16:04.453071 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.453047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" event={"ID":"617fe8fd-7289-4d0b-8384-17094658b85b","Type":"ContainerStarted","Data":"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83"} Jul 24 22:16:04.453190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.453087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" event={"ID":"617fe8fd-7289-4d0b-8384-17094658b85b","Type":"ContainerStarted","Data":"05e3229dae0eee3784114e5b959417c62b7cad18b83407d6f61ce418f678e36e"} Jul 24 22:16:04.453190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.453145 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:04.454530 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.454507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-84ckx" event={"ID":"0980d09d-775a-4f2a-82d9-7a5167813f50","Type":"ContainerStarted","Data":"fbf431ce8b58d1c6ca087214fd9778d847c097db812508ee23984609c7171ff2"} Jul 24 22:16:04.467812 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.467762 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gqshq" podStartSLOduration=67.022343184 podStartE2EDuration="1m11.467745568s" totalImagesPullingTime="4.445402384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:04.466840918 +0000 UTC m=+71.829280239" watchObservedRunningTime="2026-07-24 22:16:04.467745568 +0000 UTC m=+71.830184887" Jul 24 22:16:04.482886 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.482833 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-84ckx" podStartSLOduration=67.067905103 podStartE2EDuration="1m11.482816588s" totalImagesPullingTime="4.414911485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:04.481649752 +0000 UTC m=+71.844089073" watchObservedRunningTime="2026-07-24 22:16:04.482816588 +0000 UTC m=+71.845255910" Jul 24 22:16:04.499485 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.499428 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" podStartSLOduration=19.499406897 podStartE2EDuration="19.499406897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:04.498991351 +0000 UTC m=+71.861430674" watchObservedRunningTime="2026-07-24 22:16:04.499406897 +0000 UTC m=+71.861846219" Jul 24 22:16:04.760378 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.760260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:04.974776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.974732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:04.974969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.974800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:04.974969 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:04.974894 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:16:04.975088 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:04.974971 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls podName:d4bb5c83-712e-430d-aea1-882e82e2294d nodeName:}" failed. No retries permitted until 2026-07-24 22:16:20.974949246 +0000 UTC m=+88.337388557 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8d7hv" (UID: "d4bb5c83-712e-430d-aea1-882e82e2294d") : secret "insights-runtime-extractor-tls" not found Jul 24 22:16:04.975440 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:04.975415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4bb5c83-712e-430d-aea1-882e82e2294d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:05.168563 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:05.168463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:05.566698 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:05.566657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:05.969144 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:05.969116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:06.370047 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.369954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:06.464000 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.463968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" event={"ID":"80ac1f52-2f4f-46e7-b6e8-2d6cce205368","Type":"ContainerStarted","Data":"e5ef2542228666bc90da45bc541e7044262f91dcd4ab0b09d60bc8296da3808e"} Jul 24 22:16:06.464155 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.464024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" event={"ID":"80ac1f52-2f4f-46e7-b6e8-2d6cce205368","Type":"ContainerStarted","Data":"b62e2aa8ddb4a5b499b1daee4470f056f301e29c11b4a18e54629eebafae26fe"} Jul 24 22:16:06.481544 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.481501 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-6kcmp" podStartSLOduration=19.948461563 podStartE2EDuration="21.481486924s" totalImagesPullingTime="1.533025361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:06.480830999 +0000 UTC m=+73.843270320" watchObservedRunningTime="2026-07-24 22:16:06.481486924 +0000 UTC m=+73.843926245" Jul 24 22:16:06.763518 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.763491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:06.823121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.823096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:06.959319 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:06.959260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:07.157498 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:07.157419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:08.176215 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.176187 2576 scope.go:117] "RemoveContainer" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" Jul 24 22:16:08.475940 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.475904 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:16:08.476391 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.476373 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/1.log" Jul 24 22:16:08.476464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.476409 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" containerID="957b35ba1db2593d0e2c52fe82eb018f857ed6cf70d50efa59b8eac8b4625163" exitCode=255 Jul 24 22:16:08.476464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.476438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" event={"ID":"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240","Type":"ContainerDied","Data":"957b35ba1db2593d0e2c52fe82eb018f857ed6cf70d50efa59b8eac8b4625163"} Jul 24 22:16:08.476554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.476504 2576 scope.go:117] "RemoveContainer" containerID="8c3c7f2b0bcd8b3b1176f16bb11632dee00a9b08b08da6ba764cf008b7db1e54" Jul 24 22:16:08.476828 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:08.476804 2576 scope.go:117] "RemoveContainer" containerID="957b35ba1db2593d0e2c52fe82eb018f857ed6cf70d50efa59b8eac8b4625163" Jul 24 22:16:08.476981 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:08.476962 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-sdhsp_openshift-console-operator(8a58ca8d-ae71-448c-8fd7-0a0f1ad98240)\"" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podUID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" Jul 24 22:16:09.481877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:09.481848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:16:12.047658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:12.047626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:12.103043 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:12.103005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:15.936439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:15.936309 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:16:15.936439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:15.936370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:16:15.936862 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:15.936698 2576 scope.go:117] "RemoveContainer" containerID="957b35ba1db2593d0e2c52fe82eb018f857ed6cf70d50efa59b8eac8b4625163" Jul 24 22:16:15.936899 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:15.936864 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-sdhsp_openshift-console-operator(8a58ca8d-ae71-448c-8fd7-0a0f1ad98240)\"" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podUID="8a58ca8d-ae71-448c-8fd7-0a0f1ad98240" Jul 24 22:16:16.294358 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.294323 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:16:16.318655 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.318625 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f"] Jul 24 22:16:16.322767 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.322751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.325156 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.325129 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 22:16:16.325418 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.325405 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 22:16:16.325609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.325589 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 22:16:16.325801 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.325789 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m8b8l\"" Jul 24 22:16:16.326036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.326024 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 22:16:16.342025 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.341978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f"] Jul 24 22:16:16.359189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.359163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d0840b2-2e58-4954-9a64-a9d8c974ea18-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.359358 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.359215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkx2\" (UniqueName: \"kubernetes.io/projected/8d0840b2-2e58-4954-9a64-a9d8c974ea18-kube-api-access-plkx2\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.415030 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.414999 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bf68ccf44-m4glj"] Jul 24 22:16:16.417988 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.417973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.437121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.437097 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf68ccf44-m4glj"] Jul 24 22:16:16.442065 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.442042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:16.460204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-registry-tls\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42c230f-289f-4e01-b929-9698544e66e7-ca-trust-extracted\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-trusted-ca\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-bound-sa-token\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwv9r\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-kube-api-access-vwv9r\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-installation-pull-secrets\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-registry-certificates\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d0840b2-2e58-4954-9a64-a9d8c974ea18-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.460716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-image-registry-private-configuration\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.460716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.460620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkx2\" (UniqueName: \"kubernetes.io/projected/8d0840b2-2e58-4954-9a64-a9d8c974ea18-kube-api-access-plkx2\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.462995 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.462977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d0840b2-2e58-4954-9a64-a9d8c974ea18-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.474084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.474054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkx2\" (UniqueName: \"kubernetes.io/projected/8d0840b2-2e58-4954-9a64-a9d8c974ea18-kube-api-access-plkx2\") pod \"managed-serviceaccount-addon-agent-6cd54d8558-sl28f\" (UID: \"8d0840b2-2e58-4954-9a64-a9d8c974ea18\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.527504 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.527468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:16.561787 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.561705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-registry-certificates\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.561787 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.561743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-image-registry-private-configuration\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.561990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.561950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-registry-tls\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562046 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.561992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42c230f-289f-4e01-b929-9698544e66e7-ca-trust-extracted\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562046 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-trusted-ca\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-bound-sa-token\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwv9r\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-kube-api-access-vwv9r\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-installation-pull-secrets\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562391 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42c230f-289f-4e01-b929-9698544e66e7-ca-trust-extracted\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.562774 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.562753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-registry-certificates\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.563174 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.563155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42c230f-289f-4e01-b929-9698544e66e7-trusted-ca\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.564690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.564660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-installation-pull-secrets\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.564933 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.564918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-registry-tls\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.564972 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.564951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42c230f-289f-4e01-b929-9698544e66e7-image-registry-private-configuration\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.571592 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.571568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-bound-sa-token\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.572026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.572008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwv9r\" (UniqueName: \"kubernetes.io/projected/a42c230f-289f-4e01-b929-9698544e66e7-kube-api-access-vwv9r\") pod \"image-registry-7bf68ccf44-m4glj\" (UID: \"a42c230f-289f-4e01-b929-9698544e66e7\") " pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.629067 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.629034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:16.640502 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.640480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" Jul 24 22:16:16.725964 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.725932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:16.731023 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.730984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:16.767363 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.767230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f"] Jul 24 22:16:16.769781 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:16.769753 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d0840b2_2e58_4954_9a64_a9d8c974ea18.slice/crio-16e09f8478f4f39c3375e0319ef821c68480666b44a143936aa7cc3e124d4265 WatchSource:0}: Error finding container 16e09f8478f4f39c3375e0319ef821c68480666b44a143936aa7cc3e124d4265: Status 404 returned error can't find the container with id 16e09f8478f4f39c3375e0319ef821c68480666b44a143936aa7cc3e124d4265 Jul 24 22:16:16.859060 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:16.858987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf68ccf44-m4glj"] Jul 24 22:16:16.861819 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:16.861795 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42c230f_289f_4e01_b929_9698544e66e7.slice/crio-4ea372388f11237a0c132bf1e5a6bfcd5423457df7494b6c38dfea6a5cdac68a WatchSource:0}: Error finding container 4ea372388f11237a0c132bf1e5a6bfcd5423457df7494b6c38dfea6a5cdac68a: Status 404 returned error can't find the container with id 4ea372388f11237a0c132bf1e5a6bfcd5423457df7494b6c38dfea6a5cdac68a Jul 24 22:16:17.508682 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:17.508643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" event={"ID":"a42c230f-289f-4e01-b929-9698544e66e7","Type":"ContainerStarted","Data":"3d975df7bdda02f2e3a5bf216bdc19ec9f21ad9a43901e11c325381d15c72628"} Jul 24 22:16:17.509154 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:17.508698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" event={"ID":"a42c230f-289f-4e01-b929-9698544e66e7","Type":"ContainerStarted","Data":"4ea372388f11237a0c132bf1e5a6bfcd5423457df7494b6c38dfea6a5cdac68a"} Jul 24 22:16:17.509154 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:17.508750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:17.509811 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:17.509786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" event={"ID":"8d0840b2-2e58-4954-9a64-a9d8c974ea18","Type":"ContainerStarted","Data":"16e09f8478f4f39c3375e0319ef821c68480666b44a143936aa7cc3e124d4265"} Jul 24 22:16:17.530796 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:17.530715 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" podStartSLOduration=1.530695863 podStartE2EDuration="1.530695863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:17.52967873 +0000 UTC m=+84.892118051" watchObservedRunningTime="2026-07-24 22:16:17.530695863 +0000 UTC m=+84.893135186" Jul 24 22:16:20.522256 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:20.522217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" event={"ID":"8d0840b2-2e58-4954-9a64-a9d8c974ea18","Type":"ContainerStarted","Data":"c0e306f903773e34df5c77db942607d786f5240f6cde13818b347d75c32219b1"} Jul 24 22:16:20.538211 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:20.538154 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd54d8558-sl28f" podStartSLOduration=1.336724022 podStartE2EDuration="4.538135264s" totalImagesPullingTime="3.201411242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:20.537516971 +0000 UTC m=+87.899956305" watchObservedRunningTime="2026-07-24 22:16:20.538135264 +0000 UTC m=+87.900574585" Jul 24 22:16:20.999386 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:20.999341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:21.001834 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.001804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4bb5c83-712e-430d-aea1-882e82e2294d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d7hv\" (UID: \"d4bb5c83-712e-430d-aea1-882e82e2294d\") " pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:21.185133 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.185103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nftg\"" Jul 24 22:16:21.193126 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.193102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8d7hv" Jul 24 22:16:21.372259 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.372080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8d7hv"] Jul 24 22:16:21.374750 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:21.374721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bb5c83_712e_430d_aea1_882e82e2294d.slice/crio-128e92ca2eea63a5008ace1de332e106d019ab5bacb68eef4217973df8443862 WatchSource:0}: Error finding container 128e92ca2eea63a5008ace1de332e106d019ab5bacb68eef4217973df8443862: Status 404 returned error can't find the container with id 128e92ca2eea63a5008ace1de332e106d019ab5bacb68eef4217973df8443862 Jul 24 22:16:21.402955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.402922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:21.503982 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.503942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:21.526849 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.526762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d7hv" event={"ID":"d4bb5c83-712e-430d-aea1-882e82e2294d","Type":"ContainerStarted","Data":"128e92ca2eea63a5008ace1de332e106d019ab5bacb68eef4217973df8443862"} Jul 24 22:16:21.592650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.592609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:21.664017 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:21.663983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:22.488134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:22.488106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:22.592727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:22.592691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:24.587032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.586996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:24.622745 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.622712 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-5bgvn"] Jul 24 22:16:24.640503 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.640474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-5bgvn"] Jul 24 22:16:24.640660 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.640580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.643441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643416 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 24 22:16:24.643441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643419 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 24 22:16:24.643733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 22:16:24.643788 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643775 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c58wp\"" Jul 24 22:16:24.643999 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643948 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 22:16:24.644121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.643723 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 22:16:24.664129 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.664100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:24.732473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.732437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.732638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.732503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.732638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.732528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2810ffcd-958b-4210-b860-9239954d7df8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.732638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.732567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6hh\" (UniqueName: \"kubernetes.io/projected/2810ffcd-958b-4210-b860-9239954d7df8-kube-api-access-ng6hh\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.833640 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.833605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6hh\" (UniqueName: \"kubernetes.io/projected/2810ffcd-958b-4210-b860-9239954d7df8-kube-api-access-ng6hh\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.833806 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.833690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.833806 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.833743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.833806 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.833759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2810ffcd-958b-4210-b860-9239954d7df8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.833987 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:24.833876 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 24 22:16:24.833987 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:24.833940 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls podName:2810ffcd-958b-4210-b860-9239954d7df8 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:25.33392259 +0000 UTC m=+92.696361889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-5bgvn" (UID: "2810ffcd-958b-4210-b860-9239954d7df8") : secret "prometheus-operator-tls" not found Jul 24 22:16:24.834690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.834667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2810ffcd-958b-4210-b860-9239954d7df8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.836527 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.836503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:24.843141 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:24.843085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6hh\" (UniqueName: \"kubernetes.io/projected/2810ffcd-958b-4210-b860-9239954d7df8-kube-api-access-ng6hh\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:25.337250 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:25.337201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:25.337462 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:25.337435 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 24 22:16:25.337550 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:25.337535 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls podName:2810ffcd-958b-4210-b860-9239954d7df8 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:26.33751198 +0000 UTC m=+93.699951298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-5bgvn" (UID: "2810ffcd-958b-4210-b860-9239954d7df8") : secret "prometheus-operator-tls" not found Jul 24 22:16:25.510142 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:25.510113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:26.300314 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.300271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:26.347604 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.347580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:26.350193 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.350163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2810ffcd-958b-4210-b860-9239954d7df8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-5bgvn\" (UID: \"2810ffcd-958b-4210-b860-9239954d7df8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:26.451947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.451905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" Jul 24 22:16:26.545201 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.545116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d7hv" event={"ID":"d4bb5c83-712e-430d-aea1-882e82e2294d","Type":"ContainerStarted","Data":"dd57085546f8adaa65f9080378d8fc76c42140b8ff924f063e8465c237a83357"} Jul 24 22:16:26.579324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:26.579274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-5bgvn"] Jul 24 22:16:26.582173 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:26.582140 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2810ffcd_958b_4210_b860_9239954d7df8.slice/crio-f0f8ee79dac935f2af16efa18ddf2f4f6fe6440d3d22707774021cc05d47363c WatchSource:0}: Error finding container f0f8ee79dac935f2af16efa18ddf2f4f6fe6440d3d22707774021cc05d47363c: Status 404 returned error can't find the container with id f0f8ee79dac935f2af16efa18ddf2f4f6fe6440d3d22707774021cc05d47363c Jul 24 22:16:27.550787 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:27.550750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" event={"ID":"2810ffcd-958b-4210-b860-9239954d7df8","Type":"ContainerStarted","Data":"f0f8ee79dac935f2af16efa18ddf2f4f6fe6440d3d22707774021cc05d47363c"} Jul 24 22:16:28.877397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:28.877351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:28.933610 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:28.933576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:29.175526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:29.175462 2576 scope.go:117] "RemoveContainer" containerID="957b35ba1db2593d0e2c52fe82eb018f857ed6cf70d50efa59b8eac8b4625163" Jul 24 22:16:29.984759 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:29.984726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:30.564889 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.564851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d7hv" event={"ID":"d4bb5c83-712e-430d-aea1-882e82e2294d","Type":"ContainerStarted","Data":"932e5c4745b3ea0d2675d39b8a964077366c2a7324a875ff8a53ceacc02b848f"} Jul 24 22:16:30.567011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.566988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:16:30.567156 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.567095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" event={"ID":"8a58ca8d-ae71-448c-8fd7-0a0f1ad98240","Type":"ContainerStarted","Data":"58de434d1343ed5ac7b2b955119769631a77250ae26c06489424c37226c879ef"} Jul 24 22:16:30.567527 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.567506 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:16:30.569104 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.569078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" event={"ID":"2810ffcd-958b-4210-b860-9239954d7df8","Type":"ContainerStarted","Data":"cb16ad1f60537ff51396783d467f028b964c8a6b856946a2331a02ad75421f71"} Jul 24 22:16:30.569206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.569127 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" event={"ID":"2810ffcd-958b-4210-b860-9239954d7df8","Type":"ContainerStarted","Data":"8c07ab80453307b5eafa871b9e059eff56d120c08159f215ef24cc37f5d6fccd"} Jul 24 22:16:30.586398 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.586340 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" podStartSLOduration=43.217294913 podStartE2EDuration="45.586322612s" totalImagesPullingTime="2.369027699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:30.585676284 +0000 UTC m=+97.948115605" watchObservedRunningTime="2026-07-24 22:16:30.586322612 +0000 UTC m=+97.948761930" Jul 24 22:16:30.608086 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.608015 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-5bgvn" podStartSLOduration=3.46644024 podStartE2EDuration="6.607994122s" totalImagesPullingTime="3.141553882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:30.607867318 +0000 UTC m=+97.970306640" watchObservedRunningTime="2026-07-24 22:16:30.607994122 +0000 UTC m=+97.970433445" Jul 24 22:16:30.613031 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.613001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-sdhsp" Jul 24 22:16:30.813833 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.813796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:30.886022 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.885928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:30.945577 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:30.945547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.087431 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.087379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.268553 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.268512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.331492 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.331457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.397745 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.397712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:16:31.400609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.400580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386afff3-1e58-4e1b-95be-2cc4087b78f7-metrics-tls\") pod \"dns-default-vpljc\" (UID: \"386afff3-1e58-4e1b-95be-2cc4087b78f7\") " pod="openshift-dns/dns-default-vpljc" Jul 24 22:16:31.416338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.416283 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wnkgk\"" Jul 24 22:16:31.423865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.423838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpljc" Jul 24 22:16:31.568780 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.568692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpljc"] Jul 24 22:16:31.572931 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:31.572880 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386afff3_1e58_4e1b_95be_2cc4087b78f7.slice/crio-f3c4cae47baf90e1920189554b6dae9d167395ad178abd92ebf1b56708fc0ebc WatchSource:0}: Error finding container f3c4cae47baf90e1920189554b6dae9d167395ad178abd92ebf1b56708fc0ebc: Status 404 returned error can't find the container with id f3c4cae47baf90e1920189554b6dae9d167395ad178abd92ebf1b56708fc0ebc Jul 24 22:16:31.679082 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.679040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.740719 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.740678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.881921 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.881843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:31.982057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:31.982017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:32.302128 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.302045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:32.486040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.486005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:32.579355 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.579187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpljc" event={"ID":"386afff3-1e58-4e1b-95be-2cc4087b78f7","Type":"ContainerStarted","Data":"f3c4cae47baf90e1920189554b6dae9d167395ad178abd92ebf1b56708fc0ebc"} Jul 24 22:16:32.743220 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.743181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:32.839321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.839218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:32.978896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:32.978862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.021147 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.021106 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2f22m"] Jul 24 22:16:33.084835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.084795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.178897 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.178820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.189764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.189741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.192460 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.192435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 22:16:33.192653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.192600 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 22:16:33.192755 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.192737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gv6z9\"" Jul 24 22:16:33.192858 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.192819 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 22:16:33.196264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.195164 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn"] Jul 24 22:16:33.254042 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.254004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn"] Jul 24 22:16:33.254240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.254137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.257358 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.257334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wcl9q\"" Jul 24 22:16:33.258892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.258577 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 24 22:16:33.258892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.258597 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 24 22:16:33.258892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.258879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 24 22:16:33.316692 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-metrics-client-ca\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-sys\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-textfile\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-tls\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-root\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.316975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-wtmp\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.317138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.317000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xhk\" (UniqueName: \"kubernetes.io/projected/23330042-d32f-4094-afbf-64ff69e8ef93-kube-api-access-28xhk\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.376738 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.376702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.417918 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.417884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-wtmp\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.417918 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.417923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xhk\" (UniqueName: \"kubernetes.io/projected/23330042-d32f-4094-afbf-64ff69e8ef93-kube-api-access-28xhk\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.417968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.417997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-metrics-client-ca\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-sys\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-wtmp\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-textfile\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418161 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-sys\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-tls\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-root\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23330042-d32f-4094-afbf-64ff69e8ef93-root\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw22j\" (UniqueName: \"kubernetes.io/projected/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-api-access-jw22j\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7009454d-7544-4d3b-b842-921f3b7a55ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.418898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-textfile\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.418898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.418743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23330042-d32f-4094-afbf-64ff69e8ef93-metrics-client-ca\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.420864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.420844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-tls\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.420952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.420885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23330042-d32f-4094-afbf-64ff69e8ef93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.426250 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.426225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xhk\" (UniqueName: \"kubernetes.io/projected/23330042-d32f-4094-afbf-64ff69e8ef93-kube-api-access-28xhk\") pod \"node-exporter-2f22m\" (UID: \"23330042-d32f-4094-afbf-64ff69e8ef93\") " pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.475538 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.475509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.503638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.503601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f22m" Jul 24 22:16:33.519175 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw22j\" (UniqueName: \"kubernetes.io/projected/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-api-access-jw22j\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519483 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:33.519401 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 24 22:16:33.519483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7009454d-7544-4d3b-b842-921f3b7a55ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519483 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:33.519482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls podName:7009454d-7544-4d3b-b842-921f3b7a55ea nodeName:}" failed. No retries permitted until 2026-07-24 22:16:34.019459975 +0000 UTC m=+101.381899275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-mkmdn" (UID: "7009454d-7544-4d3b-b842-921f3b7a55ea") : secret "kube-state-metrics-tls" not found Jul 24 22:16:33.519628 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519796 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7009454d-7544-4d3b-b842-921f3b7a55ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.519994 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.519973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.520170 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.520152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009454d-7544-4d3b-b842-921f3b7a55ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.522980 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.522954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.530894 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.530867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw22j\" (UniqueName: \"kubernetes.io/projected/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-api-access-jw22j\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:33.540275 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:33.540250 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23330042_d32f_4094_afbf_64ff69e8ef93.slice/crio-5dd12d67f22e388bbcc010511c4cf19320c11da91c69b2db6ce3be847d6e25a1 WatchSource:0}: Error finding container 5dd12d67f22e388bbcc010511c4cf19320c11da91c69b2db6ce3be847d6e25a1: Status 404 returned error can't find the container with id 5dd12d67f22e388bbcc010511c4cf19320c11da91c69b2db6ce3be847d6e25a1 Jul 24 22:16:33.583667 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.583634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f22m" event={"ID":"23330042-d32f-4094-afbf-64ff69e8ef93","Type":"ContainerStarted","Data":"5dd12d67f22e388bbcc010511c4cf19320c11da91c69b2db6ce3be847d6e25a1"} Jul 24 22:16:33.626017 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.625982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.723630 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.723601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.847236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.847208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:33.980384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:33.980348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:34.023688 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.023650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:34.023908 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:34.023885 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 24 22:16:34.024134 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:34.023958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls podName:7009454d-7544-4d3b-b842-921f3b7a55ea nodeName:}" failed. No retries permitted until 2026-07-24 22:16:35.023937704 +0000 UTC m=+102.386377009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-mkmdn" (UID: "7009454d-7544-4d3b-b842-921f3b7a55ea") : secret "kube-state-metrics-tls" not found Jul 24 22:16:34.062230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.062181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:34.069160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.069073 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:16:34.091581 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.091366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:16:34.091581 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.091502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.094860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.094860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095087 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4hpr6\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095312 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095410 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095473 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 22:16:34.096084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.095656 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 22:16:34.098742 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.098590 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 22:16:34.101528 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.101236 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 22:16:34.183818 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.183736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:34.225616 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zxh\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.225854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.226218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.225993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.226218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.226019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.226218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.226087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.226218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.226109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.226218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.226156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.285663 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.285627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:34.326912 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.326912 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.326912 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxh\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.326999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.327466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.327241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.328159 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.328132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.328637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.328610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.330234 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.330206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.330397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.330362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.330485 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.330393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.330685 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.330660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.330820 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.330706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.331040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.331016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.331866 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.331836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.331997 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.331970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.332273 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.332256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.332378 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.332334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.335071 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.335054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zxh\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh\") pod \"alertmanager-main-0\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.382560 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.382519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:34.409419 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.409375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:16:34.591580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.591487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d7hv" event={"ID":"d4bb5c83-712e-430d-aea1-882e82e2294d","Type":"ContainerStarted","Data":"d374bdc262fc860f99357d909d9d6d3ce92b2c1cbc71150ddb6c09280548ee4d"} Jul 24 22:16:34.612130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:34.612078 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8d7hv" podStartSLOduration=33.297913025 podStartE2EDuration="45.612060734s" totalImagesPullingTime="12.314147709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:15:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:34.611112763 +0000 UTC m=+101.973552084" watchObservedRunningTime="2026-07-24 22:16:34.612060734 +0000 UTC m=+101.974500055" Jul 24 22:16:35.033953 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.033899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:35.036885 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.036854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7009454d-7544-4d3b-b842-921f3b7a55ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mkmdn\" (UID: \"7009454d-7544-4d3b-b842-921f3b7a55ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:35.066810 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.066769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" Jul 24 22:16:35.460975 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.460940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gqshq" Jul 24 22:16:35.597036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.596661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn"] Jul 24 22:16:35.599481 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:35.599451 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009454d_7544_4d3b_b842_921f3b7a55ea.slice/crio-c54e8addb228d5844bbc1486785741407f6cf3fa648f6c54b772c85a4452cb15 WatchSource:0}: Error finding container c54e8addb228d5844bbc1486785741407f6cf3fa648f6c54b772c85a4452cb15: Status 404 returned error can't find the container with id c54e8addb228d5844bbc1486785741407f6cf3fa648f6c54b772c85a4452cb15 Jul 24 22:16:35.599735 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.599702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f22m" event={"ID":"23330042-d32f-4094-afbf-64ff69e8ef93","Type":"ContainerStarted","Data":"4f241df4bc163bdddb44d8f7e11b07749166a932c23af6fbd4f4a0148217d7b7"} Jul 24 22:16:35.637659 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.637631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:16:35.639194 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:35.639162 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ae66e7_4a84_4c2b_950c_079d4481a948.slice/crio-a70c53ab55d64b2b95ba44cfe4413377234a503da574ca96edafdf55b6ed88de WatchSource:0}: Error finding container a70c53ab55d64b2b95ba44cfe4413377234a503da574ca96edafdf55b6ed88de: Status 404 returned error can't find the container with id a70c53ab55d64b2b95ba44cfe4413377234a503da574ca96edafdf55b6ed88de Jul 24 22:16:35.986547 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:35.986508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:36.088810 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.088743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:36.318146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.318064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:36.418423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.418361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:36.604863 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.604711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"a70c53ab55d64b2b95ba44cfe4413377234a503da574ca96edafdf55b6ed88de"} Jul 24 22:16:36.606306 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.606277 2576 generic.go:356] "Generic (PLEG): container finished" podID="23330042-d32f-4094-afbf-64ff69e8ef93" containerID="4f241df4bc163bdddb44d8f7e11b07749166a932c23af6fbd4f4a0148217d7b7" exitCode=0 Jul 24 22:16:36.606440 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.606373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f22m" event={"ID":"23330042-d32f-4094-afbf-64ff69e8ef93","Type":"ContainerDied","Data":"4f241df4bc163bdddb44d8f7e11b07749166a932c23af6fbd4f4a0148217d7b7"} Jul 24 22:16:36.608380 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.608356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpljc" event={"ID":"386afff3-1e58-4e1b-95be-2cc4087b78f7","Type":"ContainerStarted","Data":"d986a5bcc81a02d1ae641ea5db116970007501dffffac955999c72b53754d8ad"} Jul 24 22:16:36.608494 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.608403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpljc" event={"ID":"386afff3-1e58-4e1b-95be-2cc4087b78f7","Type":"ContainerStarted","Data":"b9040e137ac0ad43e0621103886bb589c7ff5dd0f9cc54f690406eeefdd94dc9"} Jul 24 22:16:36.608494 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.608453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpljc" Jul 24 22:16:36.609503 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.609482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" event={"ID":"7009454d-7544-4d3b-b842-921f3b7a55ea","Type":"ContainerStarted","Data":"c54e8addb228d5844bbc1486785741407f6cf3fa648f6c54b772c85a4452cb15"} Jul 24 22:16:36.659160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:36.659091 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vpljc" podStartSLOduration=65.786802886 podStartE2EDuration="1m9.659067466s" totalImagesPullingTime="3.87226458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:36.657483436 +0000 UTC m=+104.019923034" watchObservedRunningTime="2026-07-24 22:16:36.659067466 +0000 UTC m=+104.021506790" Jul 24 22:16:37.615213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.615113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f22m" event={"ID":"23330042-d32f-4094-afbf-64ff69e8ef93","Type":"ContainerStarted","Data":"99e3f5e23243d17c5ad89e5ce7f4af261da6869c223bb90aafa929ee48661ad7"} Jul 24 22:16:37.615213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.615203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f22m" event={"ID":"23330042-d32f-4094-afbf-64ff69e8ef93","Type":"ContainerStarted","Data":"58998d63bb808e28e8f8fbf06f7b3ffa437a2956dc7c95d03fb57ffa5c633b6e"} Jul 24 22:16:37.617049 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.617025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" event={"ID":"7009454d-7544-4d3b-b842-921f3b7a55ea","Type":"ContainerStarted","Data":"2c377f312b48649e52e342c011271eef225876b7e9e0c9988baa8e6b7ae76886"} Jul 24 22:16:37.617176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.617082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" event={"ID":"7009454d-7544-4d3b-b842-921f3b7a55ea","Type":"ContainerStarted","Data":"79db04670c22b350aca0951a9276107148fed49f88d3b5c4a2c0e761ce67e50f"} Jul 24 22:16:37.617176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.617105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" event={"ID":"7009454d-7544-4d3b-b842-921f3b7a55ea","Type":"ContainerStarted","Data":"38b63e011e63472ab1f8a6eb3231edd2e458db36676117c45759913dbc32a955"} Jul 24 22:16:37.618461 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.618439 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406" exitCode=0 Jul 24 22:16:37.618525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.618506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406"} Jul 24 22:16:37.636954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.636911 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2f22m" podStartSLOduration=3.6167899390000002 podStartE2EDuration="5.636895383s" totalImagesPullingTime="1.897855024s" totalInitContainerRuntime="122.25042ms" isStatefulPod=true podCreationTimestamp="2026-07-24 22:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:37.63598231 +0000 UTC m=+104.998421614" watchObservedRunningTime="2026-07-24 22:16:37.636895383 +0000 UTC m=+104.999334681" Jul 24 22:16:37.680727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.680678 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mkmdn" podStartSLOduration=4.234762361 podStartE2EDuration="5.68066289s" totalImagesPullingTime="1.445900529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:37.680224673 +0000 UTC m=+105.042663994" watchObservedRunningTime="2026-07-24 22:16:37.68066289 +0000 UTC m=+105.043102210" Jul 24 22:16:37.746392 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.746358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:37.847002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.846969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:37.952568 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:37.952529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:38.054023 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.053977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:38.160708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.160675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:38.518351 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.518320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bf68ccf44-m4glj" Jul 24 22:16:38.620349 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.620321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:38.721643 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.721608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:38.993734 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:38.993703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:39.095183 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.095157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:39.300825 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.300793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:39.632949 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.632866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb"} Jul 24 22:16:39.633438 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.632952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d"} Jul 24 22:16:39.633438 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.632995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f"} Jul 24 22:16:39.633438 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.633036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583"} Jul 24 22:16:39.633438 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:39.633076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082"} Jul 24 22:16:40.641425 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:40.641390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerStarted","Data":"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2"} Jul 24 22:16:40.673904 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:40.673847 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.297558793 podStartE2EDuration="6.673828225s" totalImagesPullingTime="3.355753871s" totalInitContainerRuntime="20.515561ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:40.672597595 +0000 UTC m=+108.035036927" watchObservedRunningTime="2026-07-24 22:16:40.673828225 +0000 UTC m=+108.036267546" Jul 24 22:16:41.313173 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.313122 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" podUID="617fe8fd-7289-4d0b-8384-17094658b85b" containerName="registry" containerID="cri-o://931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83" gracePeriod=30 Jul 24 22:16:41.545052 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.545029 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:41.646797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.646706 2576 generic.go:356] "Generic (PLEG): container finished" podID="617fe8fd-7289-4d0b-8384-17094658b85b" containerID="931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83" exitCode=0 Jul 24 22:16:41.646797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.646753 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" Jul 24 22:16:41.646797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.646789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" event={"ID":"617fe8fd-7289-4d0b-8384-17094658b85b","Type":"ContainerDied","Data":"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83"} Jul 24 22:16:41.647307 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.646838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5f867788-dtsds" event={"ID":"617fe8fd-7289-4d0b-8384-17094658b85b","Type":"ContainerDied","Data":"05e3229dae0eee3784114e5b959417c62b7cad18b83407d6f61ce418f678e36e"} Jul 24 22:16:41.647307 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.646858 2576 scope.go:117] "RemoveContainer" containerID="931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83" Jul 24 22:16:41.654991 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.654976 2576 scope.go:117] "RemoveContainer" containerID="931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83" Jul 24 22:16:41.655283 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:16:41.655260 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83\": container with ID starting with 931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83 not found: ID does not exist" containerID="931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83" Jul 24 22:16:41.655413 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.655287 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83"} err="failed to get container status \"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83\": rpc error: code = NotFound desc = could not find container \"931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83\": container with ID starting with 931763cc5fcac89f2c34c5fa789477feea6dc3cb869d14d155b0b159989b5f83 not found: ID does not exist" Jul 24 22:16:41.696646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696608 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696694 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8ls2\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696729 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696747 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.696876 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.696789 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token\") pod \"617fe8fd-7289-4d0b-8384-17094658b85b\" (UID: \"617fe8fd-7289-4d0b-8384-17094658b85b\") " Jul 24 22:16:41.697150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.697111 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:16:41.697371 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.697346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:16:41.699612 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.699581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2" (OuterVolumeSpecName: "kube-api-access-d8ls2") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "kube-api-access-d8ls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:16:41.699612 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.699586 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:16:41.699802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.699649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:16:41.699868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.699803 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:16:41.699868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.699856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:16:41.706141 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.706115 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "617fe8fd-7289-4d0b-8384-17094658b85b" (UID: "617fe8fd-7289-4d0b-8384-17094658b85b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797842 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-registry-certificates\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797877 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-image-registry-private-configuration\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797891 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/617fe8fd-7289-4d0b-8384-17094658b85b-ca-trust-extracted\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797905 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ls2\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-kube-api-access-d8ls2\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797921 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/617fe8fd-7289-4d0b-8384-17094658b85b-trusted-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797934 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-registry-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797948 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/617fe8fd-7289-4d0b-8384-17094658b85b-installation-pull-secrets\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.798044 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.797962 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/617fe8fd-7289-4d0b-8384-17094658b85b-bound-sa-token\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:16:41.973817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.973784 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:16:41.977493 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:41.977466 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b5f867788-dtsds"] Jul 24 22:16:42.628607 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:42.628576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:42.695147 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:42.695109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:43.180097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:43.180064 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617fe8fd-7289-4d0b-8384-17094658b85b" path="/var/lib/kubelet/pods/617fe8fd-7289-4d0b-8384-17094658b85b/volumes" Jul 24 22:16:45.097247 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:45.097208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:45.198120 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:45.198085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:45.306107 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:45.306067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:46.503860 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:46.503828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:46.560902 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:46.560871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:46.620983 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:46.620954 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpljc" Jul 24 22:16:48.303464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.303425 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:16:48.303868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.303738 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617fe8fd-7289-4d0b-8384-17094658b85b" containerName="registry" Jul 24 22:16:48.303868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.303750 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="617fe8fd-7289-4d0b-8384-17094658b85b" containerName="registry" Jul 24 22:16:48.303868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.303823 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="617fe8fd-7289-4d0b-8384-17094658b85b" containerName="registry" Jul 24 22:16:48.306414 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.306396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.309093 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.309072 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 22:16:48.309370 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.309356 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-shd2w\"" Jul 24 22:16:48.310589 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310565 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 22:16:48.310686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310603 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 22:16:48.310754 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310738 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 22:16:48.310812 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 22:16:48.310812 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310759 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 22:16:48.310902 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.310819 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 22:16:48.315445 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.315423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 22:16:48.317080 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.317057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:16:48.354313 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n225r\" (UniqueName: \"kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354639 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.354639 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.354529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455622 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n225r\" (UniqueName: \"kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455622 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.455887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.456063 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.455898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.456644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.456610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.456782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.456610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.456782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.456671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.456782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.456671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.458943 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.458919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.459066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.459046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.464497 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.464470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n225r\" (UniqueName: \"kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r\") pod \"console-f8975f4d9-6lqf6\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.616538 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.616438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:48.748505 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:48.748417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:16:48.752285 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:16:48.752254 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70793a5b_0eba_4200_8c85_565dc0898b16.slice/crio-df6af0e9343c34044e3646eae938bee41998231a4008b790c0db776fb40bb4b5 WatchSource:0}: Error finding container df6af0e9343c34044e3646eae938bee41998231a4008b790c0db776fb40bb4b5: Status 404 returned error can't find the container with id df6af0e9343c34044e3646eae938bee41998231a4008b790c0db776fb40bb4b5 Jul 24 22:16:49.130263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:49.130218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:49.187254 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:49.187219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:49.679779 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:49.679745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8975f4d9-6lqf6" event={"ID":"70793a5b-0eba-4200-8c85-565dc0898b16","Type":"ContainerStarted","Data":"df6af0e9343c34044e3646eae938bee41998231a4008b790c0db776fb40bb4b5"} Jul 24 22:16:49.903640 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:49.903609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:49.966227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:49.966193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:51.711910 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:51.711880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:52.694526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:52.694494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8975f4d9-6lqf6" event={"ID":"70793a5b-0eba-4200-8c85-565dc0898b16","Type":"ContainerStarted","Data":"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46"} Jul 24 22:16:52.718588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:52.718531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f8975f4d9-6lqf6" podStartSLOduration=1.851932765 podStartE2EDuration="4.718516604s" totalImagesPullingTime="2.866583839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:52.717942203 +0000 UTC m=+120.080381525" watchObservedRunningTime="2026-07-24 22:16:52.718516604 +0000 UTC m=+120.080955925" Jul 24 22:16:53.312798 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:53.312714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:53.423936 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:53.423892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:53.661241 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:53.661152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:53.761315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:53.761272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:53.861564 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:53.861534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:54.703419 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:54.703387 2576 generic.go:356] "Generic (PLEG): container finished" podID="33138bc3-3754-4211-85fc-ed955daaaa5f" containerID="cefb3bef7a671727daf25b315cec23a118b3f62ceb868b2bcc8447b35ec0a2ab" exitCode=0 Jul 24 22:16:54.703588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:54.703465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" event={"ID":"33138bc3-3754-4211-85fc-ed955daaaa5f","Type":"ContainerDied","Data":"cefb3bef7a671727daf25b315cec23a118b3f62ceb868b2bcc8447b35ec0a2ab"} Jul 24 22:16:54.703790 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:54.703774 2576 scope.go:117] "RemoveContainer" containerID="cefb3bef7a671727daf25b315cec23a118b3f62ceb868b2bcc8447b35ec0a2ab" Jul 24 22:16:55.708813 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:55.708781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt654" event={"ID":"33138bc3-3754-4211-85fc-ed955daaaa5f","Type":"ContainerStarted","Data":"dc1fe8f97e87aaa23e648b254ad2425e85df385e0147b9be7d0ed635624f0b4d"} Jul 24 22:16:55.761918 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:55.761883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:57.035698 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:57.035668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:57.102953 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:57.102921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:16:58.617372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:58.617336 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:58.617721 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:58.617402 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:58.622246 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:58.622223 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:16:58.723716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:16:58.723690 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:17:00.105624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:00.105595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:00.160926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:00.160901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:00.711962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:00.711931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:00.773670 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:00.773638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:01.314034 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:01.314006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:01.366967 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:01.366939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:02.984724 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:02.984688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:17:02.987171 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:02.987145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b-metrics-certs\") pod \"network-metrics-daemon-t42k6\" (UID: \"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b\") " pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:17:03.090554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:03.090524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-767d7\"" Jul 24 22:17:03.098091 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:03.098066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t42k6" Jul 24 22:17:03.550830 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:03.550805 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t42k6"] Jul 24 22:17:03.553191 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:17:03.553166 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cedf5c_1a7f_4b63_8797_f6a4f8e13c6b.slice/crio-c5298fed140f6cc502274c9c1857dac08b89df0d90fd8ab8d322d3815fb08f48 WatchSource:0}: Error finding container c5298fed140f6cc502274c9c1857dac08b89df0d90fd8ab8d322d3815fb08f48: Status 404 returned error can't find the container with id c5298fed140f6cc502274c9c1857dac08b89df0d90fd8ab8d322d3815fb08f48 Jul 24 22:17:03.740119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:03.740072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t42k6" event={"ID":"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b","Type":"ContainerStarted","Data":"c5298fed140f6cc502274c9c1857dac08b89df0d90fd8ab8d322d3815fb08f48"} Jul 24 22:17:04.745113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:04.745086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t42k6" event={"ID":"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b","Type":"ContainerStarted","Data":"97151ef54018802f2dec9705f1ce036340319ec801f1af78e2c3c6866e20e2ed"} Jul 24 22:17:05.750258 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:05.750225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t42k6" event={"ID":"a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b","Type":"ContainerStarted","Data":"2e4e1798e05856ad1dc6810931242bd957aa642ebcc95d94c1a46e3e0fa689db"} Jul 24 22:17:05.767084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:05.767038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t42k6" podStartSLOduration=131.725148587 podStartE2EDuration="2m12.767023461s" totalImagesPullingTime="1.041874874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:17:05.765886298 +0000 UTC m=+133.128325620" watchObservedRunningTime="2026-07-24 22:17:05.767023461 +0000 UTC m=+133.129462782" Jul 24 22:17:05.847157 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:05.847128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:05.945259 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:05.945229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:12.045814 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:12.045781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:12.105803 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:12.105768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:52.258703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:52.258621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:52.484511 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:52.484481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:17:53.332276 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332239 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:53.332644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332556 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="alertmanager" containerID="cri-o://14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" gracePeriod=120 Jul 24 22:17:53.332702 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332641 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-metric" containerID="cri-o://b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" gracePeriod=120 Jul 24 22:17:53.332702 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332650 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-web" containerID="cri-o://2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" gracePeriod=120 Jul 24 22:17:53.332800 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332685 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy" containerID="cri-o://42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" gracePeriod=120 Jul 24 22:17:53.332800 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332760 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="prom-label-proxy" containerID="cri-o://74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" gracePeriod=120 Jul 24 22:17:53.332800 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.332685 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="config-reloader" containerID="cri-o://7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" gracePeriod=120 Jul 24 22:17:53.930037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930006 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" exitCode=0 Jul 24 22:17:53.930037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930032 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" exitCode=0 Jul 24 22:17:53.930037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930040 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" exitCode=0 Jul 24 22:17:53.930274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930053 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" exitCode=0 Jul 24 22:17:53.930274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2"} Jul 24 22:17:53.930274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d"} Jul 24 22:17:53.930274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583"} Jul 24 22:17:53.930274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:53.930193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082"} Jul 24 22:17:54.575603 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.575578 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:54.687900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687817 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.687900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687860 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.687900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6zxh\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.687995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688059 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688134 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config\") pod \"44ae66e7-4a84-4c2b-950c-079d4481a948\" (UID: \"44ae66e7-4a84-4c2b-950c-079d4481a948\") " Jul 24 22:17:54.688696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688231 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:17:54.688696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688320 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:54.688696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688456 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-main-db\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.688696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.688477 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.689358 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.689279 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:54.691238 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.691205 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh" (OuterVolumeSpecName: "kube-api-access-d6zxh") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "kube-api-access-d6zxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:54.691584 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.691547 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume" (OuterVolumeSpecName: "config-volume") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.691708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.691686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out" (OuterVolumeSpecName: "config-out") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:17:54.691838 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.691813 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.692261 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.692234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.692407 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.692264 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.692678 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.692653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.692910 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.692892 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:54.696552 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.696390 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.702710 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.702684 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config" (OuterVolumeSpecName: "web-config") pod "44ae66e7-4a84-4c2b-950c-079d4481a948" (UID: "44ae66e7-4a84-4c2b-950c-079d4481a948"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:54.788847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788812 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44ae66e7-4a84-4c2b-950c-079d4481a948-config-out\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.788847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788842 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.788847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788854 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zxh\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-kube-api-access-d6zxh\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788864 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44ae66e7-4a84-4c2b-950c-079d4481a948-tls-assets\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788873 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788883 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-cluster-tls-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788891 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788900 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-secret-alertmanager-main-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788908 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-config-volume\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788915 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44ae66e7-4a84-4c2b-950c-079d4481a948-metrics-client-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.789081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.788942 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44ae66e7-4a84-4c2b-950c-079d4481a948-web-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:17:54.937271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937236 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" exitCode=0 Jul 24 22:17:54.937271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937264 2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerID="2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" exitCode=0 Jul 24 22:17:54.937471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb"} Jul 24 22:17:54.937471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937393 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:54.937471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f"} Jul 24 22:17:54.937471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44ae66e7-4a84-4c2b-950c-079d4481a948","Type":"ContainerDied","Data":"a70c53ab55d64b2b95ba44cfe4413377234a503da574ca96edafdf55b6ed88de"} Jul 24 22:17:54.937471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.937473 2576 scope.go:117] "RemoveContainer" containerID="74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" Jul 24 22:17:54.948969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.948947 2576 scope.go:117] "RemoveContainer" containerID="b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" Jul 24 22:17:54.955996 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.955980 2576 scope.go:117] "RemoveContainer" containerID="42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" Jul 24 22:17:54.963002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.962980 2576 scope.go:117] "RemoveContainer" containerID="2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" Jul 24 22:17:54.964418 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.964397 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:54.967909 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.967886 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:54.970413 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.970393 2576 scope.go:117] "RemoveContainer" containerID="7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" Jul 24 22:17:54.977352 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.977337 2576 scope.go:117] "RemoveContainer" containerID="14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" Jul 24 22:17:54.984208 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.984188 2576 scope.go:117] "RemoveContainer" containerID="572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406" Jul 24 22:17:54.991202 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991186 2576 scope.go:117] "RemoveContainer" containerID="74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" Jul 24 22:17:54.991477 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.991459 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2\": container with ID starting with 74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2 not found: ID does not exist" containerID="74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" Jul 24 22:17:54.991532 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991488 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2"} err="failed to get container status \"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2\": rpc error: code = NotFound desc = could not find container \"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2\": container with ID starting with 74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2 not found: ID does not exist" Jul 24 22:17:54.991532 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991507 2576 scope.go:117] "RemoveContainer" containerID="b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" Jul 24 22:17:54.991740 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.991726 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb\": container with ID starting with b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb not found: ID does not exist" containerID="b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" Jul 24 22:17:54.991772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991745 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb"} err="failed to get container status \"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb\": rpc error: code = NotFound desc = could not find container \"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb\": container with ID starting with b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb not found: ID does not exist" Jul 24 22:17:54.991772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991763 2576 scope.go:117] "RemoveContainer" containerID="42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" Jul 24 22:17:54.991973 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.991957 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d\": container with ID starting with 42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d not found: ID does not exist" containerID="42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" Jul 24 22:17:54.992009 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991978 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d"} err="failed to get container status \"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d\": rpc error: code = NotFound desc = could not find container \"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d\": container with ID starting with 42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d not found: ID does not exist" Jul 24 22:17:54.992009 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.991993 2576 scope.go:117] "RemoveContainer" containerID="2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" Jul 24 22:17:54.992236 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.992218 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f\": container with ID starting with 2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f not found: ID does not exist" containerID="2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" Jul 24 22:17:54.992286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992239 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f"} err="failed to get container status \"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f\": rpc error: code = NotFound desc = could not find container \"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f\": container with ID starting with 2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f not found: ID does not exist" Jul 24 22:17:54.992286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992251 2576 scope.go:117] "RemoveContainer" containerID="7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" Jul 24 22:17:54.992472 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.992457 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583\": container with ID starting with 7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583 not found: ID does not exist" containerID="7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" Jul 24 22:17:54.992506 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992476 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583"} err="failed to get container status \"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583\": rpc error: code = NotFound desc = could not find container \"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583\": container with ID starting with 7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583 not found: ID does not exist" Jul 24 22:17:54.992506 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992488 2576 scope.go:117] "RemoveContainer" containerID="14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" Jul 24 22:17:54.992724 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.992706 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082\": container with ID starting with 14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082 not found: ID does not exist" containerID="14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" Jul 24 22:17:54.992773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992729 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082"} err="failed to get container status \"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082\": rpc error: code = NotFound desc = could not find container \"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082\": container with ID starting with 14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082 not found: ID does not exist" Jul 24 22:17:54.992773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992745 2576 scope.go:117] "RemoveContainer" containerID="572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406" Jul 24 22:17:54.992980 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:17:54.992964 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406\": container with ID starting with 572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406 not found: ID does not exist" containerID="572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406" Jul 24 22:17:54.993021 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992984 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406"} err="failed to get container status \"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406\": rpc error: code = NotFound desc = could not find container \"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406\": container with ID starting with 572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406 not found: ID does not exist" Jul 24 22:17:54.993021 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.992997 2576 scope.go:117] "RemoveContainer" containerID="74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2" Jul 24 22:17:54.993194 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993178 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2"} err="failed to get container status \"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2\": rpc error: code = NotFound desc = could not find container \"74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2\": container with ID starting with 74873baf282fe520503b4f12e3c923903519e1221cfd5cf14e772413ff96dee2 not found: ID does not exist" Jul 24 22:17:54.993233 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993194 2576 scope.go:117] "RemoveContainer" containerID="b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb" Jul 24 22:17:54.993394 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993380 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb"} err="failed to get container status \"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb\": rpc error: code = NotFound desc = could not find container \"b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb\": container with ID starting with b1815184b447620486d4a4f8d07813fbfbf64efc7c7f30dc3b823d4cd35fc8bb not found: ID does not exist" Jul 24 22:17:54.993438 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993394 2576 scope.go:117] "RemoveContainer" containerID="42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d" Jul 24 22:17:54.993637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993620 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d"} err="failed to get container status \"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d\": rpc error: code = NotFound desc = could not find container \"42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d\": container with ID starting with 42a469cddb29a82d8a9b5db5d670ec668b3919fe38331e14177aee039b858e2d not found: ID does not exist" Jul 24 22:17:54.993681 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993637 2576 scope.go:117] "RemoveContainer" containerID="2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f" Jul 24 22:17:54.993823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993805 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f"} err="failed to get container status \"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f\": rpc error: code = NotFound desc = could not find container \"2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f\": container with ID starting with 2b88a40d65b9976bce5bb69fd2f12b00b37362881b9998fb90b2d38a6062014f not found: ID does not exist" Jul 24 22:17:54.993861 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993825 2576 scope.go:117] "RemoveContainer" containerID="7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583" Jul 24 22:17:54.993998 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993981 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583"} err="failed to get container status \"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583\": rpc error: code = NotFound desc = could not find container \"7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583\": container with ID starting with 7a876565f37f4d4d36d35a06dc83bb3e0d4ba4ab531245ad26884af40eb6e583 not found: ID does not exist" Jul 24 22:17:54.994042 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.993999 2576 scope.go:117] "RemoveContainer" containerID="14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082" Jul 24 22:17:54.994200 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.994186 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082"} err="failed to get container status \"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082\": rpc error: code = NotFound desc = could not find container \"14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082\": container with ID starting with 14933964c3bb45b002dc59f0a420ee1533e8bca1cb61e84545453f78f0826082 not found: ID does not exist" Jul 24 22:17:54.994246 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.994200 2576 scope.go:117] "RemoveContainer" containerID="572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406" Jul 24 22:17:54.994437 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.994418 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406"} err="failed to get container status \"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406\": rpc error: code = NotFound desc = could not find container \"572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406\": container with ID starting with 572b8883268d37f9a2ffae9ecf401b31047e54e38ca315b6cf92f74f71105406 not found: ID does not exist" Jul 24 22:17:54.998242 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998219 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:54.998634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998605 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-web" Jul 24 22:17:54.998634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998620 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-web" Jul 24 22:17:54.998634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998631 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="prom-label-proxy" Jul 24 22:17:54.998634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998636 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="prom-label-proxy" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998642 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="alertmanager" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998648 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="alertmanager" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998655 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="config-reloader" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998660 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="config-reloader" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998669 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="init-config-reloader" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998674 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="init-config-reloader" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998683 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-metric" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998688 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-metric" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998701 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998706 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998745 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="prom-label-proxy" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998755 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998760 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-metric" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998766 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="config-reloader" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998770 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="kube-rbac-proxy-web" Jul 24 22:17:54.998883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:54.998775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" containerName="alertmanager" Jul 24 22:17:55.003762 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.003747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.006334 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006307 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 22:17:55.006436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006341 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 22:17:55.006487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006434 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 22:17:55.006619 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 22:17:55.006757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 22:17:55.006757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006682 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 22:17:55.006757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006695 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4hpr6\"" Jul 24 22:17:55.006757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 22:17:55.006985 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.006930 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 22:17:55.011430 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.011410 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 22:17:55.015065 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.015045 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:55.091957 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.091927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.091957 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.091960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.091987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78kp\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-kube-api-access-l78kp\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-config-volume\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092160 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-web-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-config-out\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.092423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.092394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.179673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.179643 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ae66e7-4a84-4c2b-950c-079d4481a948" path="/var/lib/kubelet/pods/44ae66e7-4a84-4c2b-950c-079d4481a948/volumes" Jul 24 22:17:55.193139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78kp\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-kube-api-access-l78kp\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-config-volume\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-web-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-config-out\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.193974 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.194286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.193975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.194422 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.194375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421582e8-d17e-4ff7-9136-88e1aef78f35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.196668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.196640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.196772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.196724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.196772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.196732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.196886 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.196775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.196942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.196907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.197054 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.197035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-web-config\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.197122 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.197099 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-config-volume\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.197241 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.197221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/421582e8-d17e-4ff7-9136-88e1aef78f35-config-out\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.198601 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.198584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/421582e8-d17e-4ff7-9136-88e1aef78f35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.201180 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.201163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78kp\" (UniqueName: \"kubernetes.io/projected/421582e8-d17e-4ff7-9136-88e1aef78f35-kube-api-access-l78kp\") pod \"alertmanager-main-0\" (UID: \"421582e8-d17e-4ff7-9136-88e1aef78f35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.313189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.313153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:17:55.451840 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.451787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:17:55.453830 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:17:55.453807 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421582e8_d17e_4ff7_9136_88e1aef78f35.slice/crio-d5d92b4aaa052d7d11a4dfdba33ce9e91278c67b7df1c30e2786e180697ed9c7 WatchSource:0}: Error finding container d5d92b4aaa052d7d11a4dfdba33ce9e91278c67b7df1c30e2786e180697ed9c7: Status 404 returned error can't find the container with id d5d92b4aaa052d7d11a4dfdba33ce9e91278c67b7df1c30e2786e180697ed9c7 Jul 24 22:17:55.943706 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.943678 2576 generic.go:356] "Generic (PLEG): container finished" podID="421582e8-d17e-4ff7-9136-88e1aef78f35" containerID="561f3152b105296c4e99f11419bb484672333fda5ff1eda2b5a28fa67a9bb74a" exitCode=0 Jul 24 22:17:55.944106 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.943733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerDied","Data":"561f3152b105296c4e99f11419bb484672333fda5ff1eda2b5a28fa67a9bb74a"} Jul 24 22:17:55.944106 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:55.943780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"d5d92b4aaa052d7d11a4dfdba33ce9e91278c67b7df1c30e2786e180697ed9c7"} Jul 24 22:17:56.950801 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"7dbd087f0b9b13e42d8d6944312ebec9f149dd98d3893b71403245017f23391d"} Jul 24 22:17:56.951164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"c37eb0036e65e90f3b2948abc74090d64d093c3091ec823212b04ad724ad38d4"} Jul 24 22:17:56.951164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"4cae6f0c98241f4a15b95acf31997cfe2d9f8deaa1135e1c42e9be559dfe2ee1"} Jul 24 22:17:56.951164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"7ae5eeaa9661bb4ddc47240db387278ea46ac5b4a227b2066ecb0674208b65a3"} Jul 24 22:17:56.951164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"12ad298c20340a85991d052c43302834030cbbbbccf920c72fda338e2d13dedd"} Jul 24 22:17:56.951164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.950957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"421582e8-d17e-4ff7-9136-88e1aef78f35","Type":"ContainerStarted","Data":"5aff01b004f4d3d8a04c115d3cc613b87925b4c67e022a1c81c2d99ac4c0fe50"} Jul 24 22:17:56.981055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:56.980995 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.965613123 podStartE2EDuration="2.980979131s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.366008ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:17:56.979980596 +0000 UTC m=+184.342419958" watchObservedRunningTime="2026-07-24 22:17:56.980979131 +0000 UTC m=+184.343418445" Jul 24 22:17:57.339268 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.339164 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-db4c9fc48-bjv99"] Jul 24 22:17:57.343132 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.343111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.346508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346182 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 22:17:57.346508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346274 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wpd2c\"" Jul 24 22:17:57.346508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346321 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 22:17:57.346508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 22:17:57.346508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346466 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 22:17:57.346859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.346562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 22:17:57.352315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.351940 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 22:17:57.357729 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.357708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db4c9fc48-bjv99"] Jul 24 22:17:57.415799 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-federate-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.415799 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7j68\" (UniqueName: \"kubernetes.io/projected/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-kube-api-access-n7j68\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-metrics-client-ca\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-serving-certs-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.416019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.415979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7j68\" (UniqueName: \"kubernetes.io/projected/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-kube-api-access-n7j68\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-metrics-client-ca\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-serving-certs-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517615 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517615 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517730 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517796 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.517838 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.517824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-federate-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.518210 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.518182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-metrics-client-ca\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.518345 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.518208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-serving-certs-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.518401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.518382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.520185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.520149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.520327 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.520258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-federate-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.520327 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.520283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-telemeter-client-tls\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.520327 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.520320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.527722 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.527688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7j68\" (UniqueName: \"kubernetes.io/projected/42ed3bcb-99ad-4b62-ac6f-5a89af7361da-kube-api-access-n7j68\") pod \"telemeter-client-db4c9fc48-bjv99\" (UID: \"42ed3bcb-99ad-4b62-ac6f-5a89af7361da\") " pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.655523 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.655434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" Jul 24 22:17:57.798782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.798748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db4c9fc48-bjv99"] Jul 24 22:17:57.804606 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:17:57.804577 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ed3bcb_99ad_4b62_ac6f_5a89af7361da.slice/crio-0bff1ebba7a9b042a1e4c59d49c5c924347e5d7050dfaafb3ae1709dbe76d93a WatchSource:0}: Error finding container 0bff1ebba7a9b042a1e4c59d49c5c924347e5d7050dfaafb3ae1709dbe76d93a: Status 404 returned error can't find the container with id 0bff1ebba7a9b042a1e4c59d49c5c924347e5d7050dfaafb3ae1709dbe76d93a Jul 24 22:17:57.956798 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:57.956765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" event={"ID":"42ed3bcb-99ad-4b62-ac6f-5a89af7361da","Type":"ContainerStarted","Data":"0bff1ebba7a9b042a1e4c59d49c5c924347e5d7050dfaafb3ae1709dbe76d93a"} Jul 24 22:17:59.967000 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:59.966963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" event={"ID":"42ed3bcb-99ad-4b62-ac6f-5a89af7361da","Type":"ContainerStarted","Data":"a7a498ab923b016b9a39b37eb33d0d0710a63a36783b387a155073a71ff53149"} Jul 24 22:17:59.967416 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:59.967052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" event={"ID":"42ed3bcb-99ad-4b62-ac6f-5a89af7361da","Type":"ContainerStarted","Data":"c01c0a32b91fd8c5ed25c6eb273eca215a9bf0c44f94e8f47f46ef27aedbdb01"} Jul 24 22:17:59.967416 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:59.967071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" event={"ID":"42ed3bcb-99ad-4b62-ac6f-5a89af7361da","Type":"ContainerStarted","Data":"5aaad8aaf070d247a1f1cded718bf3f6773ab148faa3682dd24eab0e57c362ed"} Jul 24 22:17:59.991428 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:17:59.991379 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-db4c9fc48-bjv99" podStartSLOduration=1.4560162220000001 podStartE2EDuration="2.991364241s" totalImagesPullingTime="1.535348019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:17:59.989134801 +0000 UTC m=+187.351574123" watchObservedRunningTime="2026-07-24 22:17:59.991364241 +0000 UTC m=+187.353803608" Jul 24 22:18:00.766975 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.766945 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:18:00.770476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.770455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.782794 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.782770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:18:00.833132 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.833105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:00.846224 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.846567 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.846380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864sc\" (UniqueName: \"kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947313 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947313 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864sc\" (UniqueName: \"kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.947537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.947477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.948223 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.948187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.948223 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.948189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.948472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.948449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.948507 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.948487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.949974 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.949954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.950053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.950032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.955082 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.955059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-864sc\" (UniqueName: \"kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc\") pod \"console-76c4664b4d-q4rz5\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:00.957825 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:00.957804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:01.059603 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.059524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:01.080178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.080149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:01.220330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.220277 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:18:01.223906 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:18:01.223878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b11f0e0_575d_41e1_a968_49f2a32b98f7.slice/crio-65380e8b7317a4544e1304234bbd10615b6125dd3ed5015362e512701827c99a WatchSource:0}: Error finding container 65380e8b7317a4544e1304234bbd10615b6125dd3ed5015362e512701827c99a: Status 404 returned error can't find the container with id 65380e8b7317a4544e1304234bbd10615b6125dd3ed5015362e512701827c99a Jul 24 22:18:01.555996 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.555967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:01.608530 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.608501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:01.976648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.976609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c4664b4d-q4rz5" event={"ID":"1b11f0e0-575d-41e1-a968-49f2a32b98f7","Type":"ContainerStarted","Data":"a36c198371616528bf5931f03e9888e5ffdd78aeb8c9d0f4929a7089c45ecc57"} Jul 24 22:18:01.976783 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.976654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c4664b4d-q4rz5" event={"ID":"1b11f0e0-575d-41e1-a968-49f2a32b98f7","Type":"ContainerStarted","Data":"65380e8b7317a4544e1304234bbd10615b6125dd3ed5015362e512701827c99a"} Jul 24 22:18:01.995793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:01.995748 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76c4664b4d-q4rz5" podStartSLOduration=1.995732324 podStartE2EDuration="1.995732324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:18:01.994323984 +0000 UTC m=+189.356763308" watchObservedRunningTime="2026-07-24 22:18:01.995732324 +0000 UTC m=+189.358171644" Jul 24 22:18:11.080805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:11.080760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:11.081265 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:11.080831 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:11.085585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:11.085560 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:12.020111 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:12.020058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:18:12.062956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:12.062922 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:18:12.169439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:12.169409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:12.272712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:12.272606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:12.324850 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:12.324812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:18:37.083275 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.083224 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f8975f4d9-6lqf6" podUID="70793a5b-0eba-4200-8c85-565dc0898b16" containerName="console" containerID="cri-o://ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46" gracePeriod=15 Jul 24 22:18:37.327529 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.327502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:18:37.360923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360848 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.360923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360879 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n225r\" (UniqueName: \"kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.360923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360896 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.360923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360912 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.361222 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.361222 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.360987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.361222 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.361026 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert\") pod \"70793a5b-0eba-4200-8c85-565dc0898b16\" (UID: \"70793a5b-0eba-4200-8c85-565dc0898b16\") " Jul 24 22:18:37.361466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.361436 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca" (OuterVolumeSpecName: "service-ca") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:18:37.361532 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.361445 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config" (OuterVolumeSpecName: "console-config") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:18:37.361651 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.361527 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:18:37.361651 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.361572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:18:37.363531 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.363500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:18:37.363640 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.363602 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:18:37.363708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.363669 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r" (OuterVolumeSpecName: "kube-api-access-n225r") pod "70793a5b-0eba-4200-8c85-565dc0898b16" (UID: "70793a5b-0eba-4200-8c85-565dc0898b16"). InnerVolumeSpecName "kube-api-access-n225r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:18:37.462008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.461970 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-oauth-serving-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.461998 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-console-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.462007 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n225r\" (UniqueName: \"kubernetes.io/projected/70793a5b-0eba-4200-8c85-565dc0898b16-kube-api-access-n225r\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.462016 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462273 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.462024 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-oauth-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462273 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.462032 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70793a5b-0eba-4200-8c85-565dc0898b16-trusted-ca-bundle\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:37.462273 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:37.462040 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70793a5b-0eba-4200-8c85-565dc0898b16-console-serving-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:18:38.115411 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.115375 2576 generic.go:356] "Generic (PLEG): container finished" podID="70793a5b-0eba-4200-8c85-565dc0898b16" containerID="ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46" exitCode=0 Jul 24 22:18:38.115867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.115424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8975f4d9-6lqf6" event={"ID":"70793a5b-0eba-4200-8c85-565dc0898b16","Type":"ContainerDied","Data":"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46"} Jul 24 22:18:38.115867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.115435 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8975f4d9-6lqf6" Jul 24 22:18:38.115867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.115506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8975f4d9-6lqf6" event={"ID":"70793a5b-0eba-4200-8c85-565dc0898b16","Type":"ContainerDied","Data":"df6af0e9343c34044e3646eae938bee41998231a4008b790c0db776fb40bb4b5"} Jul 24 22:18:38.115867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.115534 2576 scope.go:117] "RemoveContainer" containerID="ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46" Jul 24 22:18:38.124383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.124363 2576 scope.go:117] "RemoveContainer" containerID="ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46" Jul 24 22:18:38.124643 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:18:38.124625 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46\": container with ID starting with ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46 not found: ID does not exist" containerID="ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46" Jul 24 22:18:38.124707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.124656 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46"} err="failed to get container status \"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46\": rpc error: code = NotFound desc = could not find container \"ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46\": container with ID starting with ebe09ba9f8220736a124aa8d35549bae995722206c18dff28a2aeba67d8aeb46 not found: ID does not exist" Jul 24 22:18:38.137920 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.137894 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:18:38.141339 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:38.141317 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8975f4d9-6lqf6"] Jul 24 22:18:39.180384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:18:39.180354 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70793a5b-0eba-4200-8c85-565dc0898b16" path="/var/lib/kubelet/pods/70793a5b-0eba-4200-8c85-565dc0898b16/volumes" Jul 24 22:19:07.480619 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:07.480587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:07.550263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:07.550231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:15.430406 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:15.430323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:19.417661 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:19.417630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:19.497041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:19.497007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:25.698521 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:25.698487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:27.050962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:27.050923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:27.153155 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:27.153115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:27.953874 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:27.953844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:30.900110 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:30.900070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:30.970183 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:30.970150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:41.248424 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:41.248394 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:19:41.384548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:41.384521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:41.484392 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:41.484358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:41.538893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:41.538813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:19:53.018810 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:53.018779 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:19:53.021617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:53.021594 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:19:53.026887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:53.026861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:19:53.030025 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:53.030000 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 22:19:53.030151 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:19:53.030135 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:20:00.065317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:00.065274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:00.165432 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:00.165399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:06.268904 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.268861 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76c4664b4d-q4rz5" podUID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" containerName="console" containerID="cri-o://a36c198371616528bf5931f03e9888e5ffdd78aeb8c9d0f4929a7089c45ecc57" gracePeriod=15 Jul 24 22:20:06.449637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.449608 2576 generic.go:356] "Generic (PLEG): container finished" podID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" containerID="a36c198371616528bf5931f03e9888e5ffdd78aeb8c9d0f4929a7089c45ecc57" exitCode=0 Jul 24 22:20:06.449790 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.449689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c4664b4d-q4rz5" event={"ID":"1b11f0e0-575d-41e1-a968-49f2a32b98f7","Type":"ContainerDied","Data":"a36c198371616528bf5931f03e9888e5ffdd78aeb8c9d0f4929a7089c45ecc57"} Jul 24 22:20:06.509191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.509164 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:20:06.582002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.581910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.581954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.581978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582002 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-864sc\" (UniqueName: \"kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582214 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config\") pod \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\" (UID: \"1b11f0e0-575d-41e1-a968-49f2a32b98f7\") " Jul 24 22:20:06.582496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582423 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config" (OuterVolumeSpecName: "console-config") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:20:06.582496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:20:06.582496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582458 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:20:06.582496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:20:06.582652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582552 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-oauth-serving-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.582652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582567 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-trusted-ca-bundle\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.582652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582576 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.582652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.582584 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b11f0e0-575d-41e1-a968-49f2a32b98f7-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.584487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.584460 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc" (OuterVolumeSpecName: "kube-api-access-864sc") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "kube-api-access-864sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:20:06.584588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.584463 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:20:06.584628 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.584584 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b11f0e0-575d-41e1-a968-49f2a32b98f7" (UID: "1b11f0e0-575d-41e1-a968-49f2a32b98f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:20:06.683324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.683276 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-serving-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.683324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.683323 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864sc\" (UniqueName: \"kubernetes.io/projected/1b11f0e0-575d-41e1-a968-49f2a32b98f7-kube-api-access-864sc\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:06.683324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:06.683334 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b11f0e0-575d-41e1-a968-49f2a32b98f7-console-oauth-config\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:20:07.455226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:07.455193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c4664b4d-q4rz5" event={"ID":"1b11f0e0-575d-41e1-a968-49f2a32b98f7","Type":"ContainerDied","Data":"65380e8b7317a4544e1304234bbd10615b6125dd3ed5015362e512701827c99a"} Jul 24 22:20:07.455674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:07.455232 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c4664b4d-q4rz5" Jul 24 22:20:07.455674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:07.455252 2576 scope.go:117] "RemoveContainer" containerID="a36c198371616528bf5931f03e9888e5ffdd78aeb8c9d0f4929a7089c45ecc57" Jul 24 22:20:07.473102 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:07.473076 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:20:07.476664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:07.476646 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76c4664b4d-q4rz5"] Jul 24 22:20:09.180546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:09.180513 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" path="/var/lib/kubelet/pods/1b11f0e0-575d-41e1-a968-49f2a32b98f7/volumes" Jul 24 22:20:26.681629 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:26.681594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:27.745940 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:27.745908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:28.632032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.631998 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78"] Jul 24 22:20:28.632319 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632290 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70793a5b-0eba-4200-8c85-565dc0898b16" containerName="console" Jul 24 22:20:28.632319 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632316 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70793a5b-0eba-4200-8c85-565dc0898b16" containerName="console" Jul 24 22:20:28.632397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632325 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" containerName="console" Jul 24 22:20:28.632397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632331 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" containerName="console" Jul 24 22:20:28.632397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632382 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b11f0e0-575d-41e1-a968-49f2a32b98f7" containerName="console" Jul 24 22:20:28.632397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.632391 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70793a5b-0eba-4200-8c85-565dc0898b16" containerName="console" Jul 24 22:20:28.635081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.635065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.638279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.638258 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2qnfb\"" Jul 24 22:20:28.638791 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.638765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:20:28.638896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.638773 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 22:20:28.655860 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.655831 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78"] Jul 24 22:20:28.657706 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.657684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqtr\" (UniqueName: \"kubernetes.io/projected/49414c29-83b0-4ae9-9387-b8db54776c6c-kube-api-access-qlqtr\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.657807 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.657729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49414c29-83b0-4ae9-9387-b8db54776c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.758604 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.758566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqtr\" (UniqueName: \"kubernetes.io/projected/49414c29-83b0-4ae9-9387-b8db54776c6c-kube-api-access-qlqtr\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.759050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.758632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49414c29-83b0-4ae9-9387-b8db54776c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.759112 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.759045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49414c29-83b0-4ae9-9387-b8db54776c6c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.767101 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.767078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqtr\" (UniqueName: \"kubernetes.io/projected/49414c29-83b0-4ae9-9387-b8db54776c6c-kube-api-access-qlqtr\") pod \"cert-manager-operator-controller-manager-55c444bf86-ccp78\" (UID: \"49414c29-83b0-4ae9-9387-b8db54776c6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:28.944420 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:28.944389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" Jul 24 22:20:29.081398 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:29.081367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78"] Jul 24 22:20:29.085139 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:20:29.085112 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49414c29_83b0_4ae9_9387_b8db54776c6c.slice/crio-53cbfaeaecf019056d137c1b8be1bae024af9c987c90644dbd4b92085f4c3457 WatchSource:0}: Error finding container 53cbfaeaecf019056d137c1b8be1bae024af9c987c90644dbd4b92085f4c3457: Status 404 returned error can't find the container with id 53cbfaeaecf019056d137c1b8be1bae024af9c987c90644dbd4b92085f4c3457 Jul 24 22:20:29.088099 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:29.088082 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:20:29.538685 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:29.538647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" event={"ID":"49414c29-83b0-4ae9-9387-b8db54776c6c","Type":"ContainerStarted","Data":"53cbfaeaecf019056d137c1b8be1bae024af9c987c90644dbd4b92085f4c3457"} Jul 24 22:20:30.297446 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:30.297408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:30.397018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:30.396976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:32.555835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:32.555780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" event={"ID":"49414c29-83b0-4ae9-9387-b8db54776c6c","Type":"ContainerStarted","Data":"21f02518b78d1ad3057abec128a5f9d0ad9435aedf802d0123f80ebebc6c0c59"} Jul 24 22:20:32.580054 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:32.580004 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ccp78" podStartSLOduration=2.03706259 podStartE2EDuration="4.579986669s" totalImagesPullingTime="2.542924079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:20:32.578815383 +0000 UTC m=+339.941254704" watchObservedRunningTime="2026-07-24 22:20:32.579986669 +0000 UTC m=+339.942425987" Jul 24 22:20:32.655664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:32.655630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:32.747786 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:32.747752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:32.966079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:32.966041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:33.077420 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:33.077387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:34.821118 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:34.821086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:35.233168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.233140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:35.296056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.296023 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh"] Jul 24 22:20:35.299412 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.299395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.301853 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.301828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2xdbn\"" Jul 24 22:20:35.301990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.301891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 22:20:35.302977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.302961 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 22:20:35.308921 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.308902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh"] Jul 24 22:20:35.412306 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.412268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5n8v\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-kube-api-access-f5n8v\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.412456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.412356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.513634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.513550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5n8v\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-kube-api-access-f5n8v\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.513634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.513600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.521944 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.521917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.522071 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.522052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5n8v\" (UniqueName: \"kubernetes.io/projected/55f6c7b1-373e-4d7b-8a51-5a9225f10205-kube-api-access-f5n8v\") pod \"cert-manager-cainjector-559cdd8c87-8tfkh\" (UID: \"55f6c7b1-373e-4d7b-8a51-5a9225f10205\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.608872 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.608835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" Jul 24 22:20:35.734413 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:35.734249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh"] Jul 24 22:20:35.736540 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:20:35.736510 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55f6c7b1_373e_4d7b_8a51_5a9225f10205.slice/crio-83bc9096b33d46d39d62f69c00d5c7ee6ef0a32bb57bffddc4a77f945144087c WatchSource:0}: Error finding container 83bc9096b33d46d39d62f69c00d5c7ee6ef0a32bb57bffddc4a77f945144087c: Status 404 returned error can't find the container with id 83bc9096b33d46d39d62f69c00d5c7ee6ef0a32bb57bffddc4a77f945144087c Jul 24 22:20:36.574041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:36.573990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" event={"ID":"55f6c7b1-373e-4d7b-8a51-5a9225f10205","Type":"ContainerStarted","Data":"83bc9096b33d46d39d62f69c00d5c7ee6ef0a32bb57bffddc4a77f945144087c"} Jul 24 22:20:37.284136 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:37.284085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:37.345279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:37.345249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:38.584500 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:38.584447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" event={"ID":"55f6c7b1-373e-4d7b-8a51-5a9225f10205","Type":"ContainerStarted","Data":"f77adadea12fb1a78f1373bb878c11c1ab665997939662c187a675439816f325"} Jul 24 22:20:38.600524 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:38.600471 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tfkh" podStartSLOduration=0.894127431 podStartE2EDuration="3.600456313s" totalImagesPullingTime="2.706328882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:20:38.599476646 +0000 UTC m=+345.961915967" watchObservedRunningTime="2026-07-24 22:20:38.600456313 +0000 UTC m=+345.962895634" Jul 24 22:20:40.837737 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.837701 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cjpmt"] Jul 24 22:20:40.840914 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.840894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:40.843363 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.843340 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h9zf7\"" Jul 24 22:20:40.849432 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.849411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cjpmt"] Jul 24 22:20:40.961423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.961385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:40.961607 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:40.961458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srf7c\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-kube-api-access-srf7c\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.062891 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.062853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srf7c\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-kube-api-access-srf7c\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.062891 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.062898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.071130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.071100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.071207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.071152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srf7c\" (UniqueName: \"kubernetes.io/projected/b4b184e0-dba7-4270-a03e-46341ed040cf-kube-api-access-srf7c\") pod \"cert-manager-webhook-6486ffd997-cjpmt\" (UID: \"b4b184e0-dba7-4270-a03e-46341ed040cf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.150237 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.150142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.283057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.283000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cjpmt"] Jul 24 22:20:41.285591 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:20:41.285563 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4b184e0_dba7_4270_a03e_46341ed040cf.slice/crio-2fc42fc0ee5622256e7503a8d5c9409a26682ca81545ad87b726a2496575f598 WatchSource:0}: Error finding container 2fc42fc0ee5622256e7503a8d5c9409a26682ca81545ad87b726a2496575f598: Status 404 returned error can't find the container with id 2fc42fc0ee5622256e7503a8d5c9409a26682ca81545ad87b726a2496575f598 Jul 24 22:20:41.598201 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.598157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" event={"ID":"b4b184e0-dba7-4270-a03e-46341ed040cf","Type":"ContainerStarted","Data":"7ea0590ca2b3fcc31fee620fe7123a3c8cb24c5d4e1277719a4c19c74ce54cef"} Jul 24 22:20:41.598395 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.598215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" event={"ID":"b4b184e0-dba7-4270-a03e-46341ed040cf","Type":"ContainerStarted","Data":"2fc42fc0ee5622256e7503a8d5c9409a26682ca81545ad87b726a2496575f598"} Jul 24 22:20:41.598395 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.598237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:41.613253 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:41.613197 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" podStartSLOduration=1.613176931 podStartE2EDuration="1.613176931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:20:41.612704476 +0000 UTC m=+348.975143798" watchObservedRunningTime="2026-07-24 22:20:41.613176931 +0000 UTC m=+348.975616254" Jul 24 22:20:43.232423 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.232391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:43.878733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.878701 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp2c2"] Jul 24 22:20:43.881845 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.881829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:43.884178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.884159 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tb54v\"" Jul 24 22:20:43.891549 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.891523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp2c2"] Jul 24 22:20:43.987100 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.987059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:43.987100 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:43.987099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tjp\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-kube-api-access-w2tjp\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.088637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.088589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.088637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.088634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tjp\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-kube-api-access-w2tjp\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.097056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.097022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-bound-sa-token\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.097193 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.097175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tjp\" (UniqueName: \"kubernetes.io/projected/e9437311-ca10-43a6-8f62-b6e32a7cf6d6-kube-api-access-w2tjp\") pod \"cert-manager-55b4d954d5-rp2c2\" (UID: \"e9437311-ca10-43a6-8f62-b6e32a7cf6d6\") " pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.191830 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.191803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rp2c2" Jul 24 22:20:44.319756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.319671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rp2c2"] Jul 24 22:20:44.322037 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:20:44.322008 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9437311_ca10_43a6_8f62_b6e32a7cf6d6.slice/crio-86472144dfdbd76848ef5da6d88a54e04599334daef8d42310e6df26de1397f0 WatchSource:0}: Error finding container 86472144dfdbd76848ef5da6d88a54e04599334daef8d42310e6df26de1397f0: Status 404 returned error can't find the container with id 86472144dfdbd76848ef5da6d88a54e04599334daef8d42310e6df26de1397f0 Jul 24 22:20:44.611747 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.611651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rp2c2" event={"ID":"e9437311-ca10-43a6-8f62-b6e32a7cf6d6","Type":"ContainerStarted","Data":"99c822b56dacff8922cf310d674451e51e9cfadd60f321903e0b35203113eae8"} Jul 24 22:20:44.611747 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.611696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rp2c2" event={"ID":"e9437311-ca10-43a6-8f62-b6e32a7cf6d6","Type":"ContainerStarted","Data":"86472144dfdbd76848ef5da6d88a54e04599334daef8d42310e6df26de1397f0"} Jul 24 22:20:44.628468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.628421 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rp2c2" podStartSLOduration=1.628407128 podStartE2EDuration="1.628407128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:20:44.626473437 +0000 UTC m=+351.988912758" watchObservedRunningTime="2026-07-24 22:20:44.628407128 +0000 UTC m=+351.990846449" Jul 24 22:20:44.952652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:44.952617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:45.054194 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:45.054163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:47.605246 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:47.605217 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cjpmt" Jul 24 22:20:51.111566 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:51.111517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:51.214112 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:51.214076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:20:52.975397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:20:52.975368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:05.711169 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:05.711131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:05.810181 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:05.810144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.337049 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.337010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.544508 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.544476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.644227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.644111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.709096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.709062 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j"] Jul 24 22:21:06.715128 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.715107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.725707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.725688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 22:21:06.725828 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.725688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 22:21:06.727326 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.727308 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 22:21:06.727805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.727791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v4rcb\"" Jul 24 22:21:06.728474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.728457 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 22:21:06.741944 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.741918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j"] Jul 24 22:21:06.882253 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.882216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.882456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.882262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4spr\" (UniqueName: \"kubernetes.io/projected/460371d3-e56e-407b-b6e0-b03f31a2e380-kube-api-access-d4spr\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.882456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.882367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.882456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.882404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.882456 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.882425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.890781 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.890759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.983068 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.983264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.983264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.983264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.983264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4spr\" (UniqueName: \"kubernetes.io/projected/460371d3-e56e-407b-b6e0-b03f31a2e380-kube-api-access-d4spr\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.983699 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.983673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.986077 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.986052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460371d3-e56e-407b-b6e0-b03f31a2e380-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.986287 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.986266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.986353 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.986342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/460371d3-e56e-407b-b6e0-b03f31a2e380-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:06.991189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.991167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:06.991664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:06.991646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4spr\" (UniqueName: \"kubernetes.io/projected/460371d3-e56e-407b-b6e0-b03f31a2e380-kube-api-access-d4spr\") pod \"opendatahub-operator-controller-manager-854fb8676-6px8j\" (UID: \"460371d3-e56e-407b-b6e0-b03f31a2e380\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:07.026227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.026193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:07.091823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.091775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:07.161014 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.160991 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j"] Jul 24 22:21:07.164432 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:07.164389 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460371d3_e56e_407b_b6e0_b03f31a2e380.slice/crio-037ccb50d205d6cd1fc0529bd3f8bede5fb7caaf2c820c4ad22166fff9d40c8b WatchSource:0}: Error finding container 037ccb50d205d6cd1fc0529bd3f8bede5fb7caaf2c820c4ad22166fff9d40c8b: Status 404 returned error can't find the container with id 037ccb50d205d6cd1fc0529bd3f8bede5fb7caaf2c820c4ad22166fff9d40c8b Jul 24 22:21:07.164432 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.164412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:07.231441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.231403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:07.331796 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.331708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:07.431426 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.431391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:07.711221 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:07.711184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" event={"ID":"460371d3-e56e-407b-b6e0-b03f31a2e380","Type":"ContainerStarted","Data":"037ccb50d205d6cd1fc0529bd3f8bede5fb7caaf2c820c4ad22166fff9d40c8b"} Jul 24 22:21:10.510526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:10.510412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:10.600026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:10.599987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:10.729643 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:10.729610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" event={"ID":"460371d3-e56e-407b-b6e0-b03f31a2e380","Type":"ContainerStarted","Data":"d32bd2a14303605c08b6d685c30187918f43746523d2c8b4512fbf7ee617e36f"} Jul 24 22:21:10.773058 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:10.772986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:10.874171 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:10.874139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:11.735992 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:11.735951 2576 generic.go:356] "Generic (PLEG): container finished" podID="460371d3-e56e-407b-b6e0-b03f31a2e380" containerID="d32bd2a14303605c08b6d685c30187918f43746523d2c8b4512fbf7ee617e36f" exitCode=0 Jul 24 22:21:11.736383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:11.736039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" event={"ID":"460371d3-e56e-407b-b6e0-b03f31a2e380","Type":"ContainerDied","Data":"d32bd2a14303605c08b6d685c30187918f43746523d2c8b4512fbf7ee617e36f"} Jul 24 22:21:12.748272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:12.748236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" event={"ID":"460371d3-e56e-407b-b6e0-b03f31a2e380","Type":"ContainerStarted","Data":"f254a7608554d00fec20a5e47dd0d48bf6e4e08fbe4ee4db23b88f1696679561"} Jul 24 22:21:12.748735 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:12.748414 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:12.770402 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:12.770352 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" podStartSLOduration=2.379666365 podStartE2EDuration="6.770336444s" totalImagesPullingTime="3.357595508s" totalInitContainerRuntime="1.033074571s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:12.768653602 +0000 UTC m=+380.131092935" watchObservedRunningTime="2026-07-24 22:21:12.770336444 +0000 UTC m=+380.132775765" Jul 24 22:21:19.168637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:19.168609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:19.270661 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:19.270628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:23.522986 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:23.522950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:23.576896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:23.576867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:23.755401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:23.755371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-6px8j" Jul 24 22:21:25.327517 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.327485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:25.460937 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.460905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:25.790031 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.789994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:25.919926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.919895 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-q748r"] Jul 24 22:21:25.926675 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.926652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:25.929383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.929362 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 22:21:25.929673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.929655 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 22:21:25.929721 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.929663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 22:21:25.929830 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.929818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 22:21:25.938452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.938430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-q748r"] Jul 24 22:21:25.943947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.943921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3e254a-21fb-4fd9-b6d9-502086724a94-tls-certs\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:25.944058 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.943967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3e254a-21fb-4fd9-b6d9-502086724a94-tmp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:25.944103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:25.944054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvgp\" (UniqueName: \"kubernetes.io/projected/7c3e254a-21fb-4fd9-b6d9-502086724a94-kube-api-access-tmvgp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.030253 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.030213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:26.045103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.045033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3e254a-21fb-4fd9-b6d9-502086724a94-tls-certs\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.045103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.045071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3e254a-21fb-4fd9-b6d9-502086724a94-tmp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.045269 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.045113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvgp\" (UniqueName: \"kubernetes.io/projected/7c3e254a-21fb-4fd9-b6d9-502086724a94-kube-api-access-tmvgp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.047499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.047473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3e254a-21fb-4fd9-b6d9-502086724a94-tmp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.047695 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.047677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3e254a-21fb-4fd9-b6d9-502086724a94-tls-certs\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.053836 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.053812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvgp\" (UniqueName: \"kubernetes.io/projected/7c3e254a-21fb-4fd9-b6d9-502086724a94-kube-api-access-tmvgp\") pod \"kube-auth-proxy-7d58b94969-q748r\" (UID: \"7c3e254a-21fb-4fd9-b6d9-502086724a94\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.091375 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.091344 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m82cl"] Jul 24 22:21:26.094778 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.094762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.097371 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.097351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xhb9f\"" Jul 24 22:21:26.097451 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.097379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 22:21:26.102810 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.102785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m82cl"] Jul 24 22:21:26.136915 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.136856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:26.145603 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.145569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclkl\" (UniqueName: \"kubernetes.io/projected/ee29d1d3-691b-4191-aecc-54a0ae5499a7-kube-api-access-xclkl\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.145769 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.145638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee29d1d3-691b-4191-aecc-54a0ae5499a7-cert\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.235259 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.235229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:26.236717 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.236698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" Jul 24 22:21:26.246825 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.246775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclkl\" (UniqueName: \"kubernetes.io/projected/ee29d1d3-691b-4191-aecc-54a0ae5499a7-kube-api-access-xclkl\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.246985 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.246832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee29d1d3-691b-4191-aecc-54a0ae5499a7-cert\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.249635 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.249605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee29d1d3-691b-4191-aecc-54a0ae5499a7-cert\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.258635 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.258610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclkl\" (UniqueName: \"kubernetes.io/projected/ee29d1d3-691b-4191-aecc-54a0ae5499a7-kube-api-access-xclkl\") pod \"odh-model-controller-7998c67dc4-m82cl\" (UID: \"ee29d1d3-691b-4191-aecc-54a0ae5499a7\") " pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.335135 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.335109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:26.372270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.372246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-q748r"] Jul 24 22:21:26.374308 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:26.374265 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3e254a_21fb_4fd9_b6d9_502086724a94.slice/crio-17ec4be4fa0f879d884f74bcff2f0967ca31304863864dab653b291d501dd595 WatchSource:0}: Error finding container 17ec4be4fa0f879d884f74bcff2f0967ca31304863864dab653b291d501dd595: Status 404 returned error can't find the container with id 17ec4be4fa0f879d884f74bcff2f0967ca31304863864dab653b291d501dd595 Jul 24 22:21:26.406083 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.406056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:26.542459 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.542438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m82cl"] Jul 24 22:21:26.544455 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:26.544424 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee29d1d3_691b_4191_aecc_54a0ae5499a7.slice/crio-36f6b30e98a7760e87220f1d9eaf0d76a709889338086ea974758f0a97ba0640 WatchSource:0}: Error finding container 36f6b30e98a7760e87220f1d9eaf0d76a709889338086ea974758f0a97ba0640: Status 404 returned error can't find the container with id 36f6b30e98a7760e87220f1d9eaf0d76a709889338086ea974758f0a97ba0640 Jul 24 22:21:26.656121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.656038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:26.669709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.669671 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mm5jq"] Jul 24 22:21:26.674668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.674644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:26.677879 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.677849 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 22:21:26.678001 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.677978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-msj65\"" Jul 24 22:21:26.690570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.690547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mm5jq"] Jul 24 22:21:26.751121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.751090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:26.751315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.751128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x98x\" (UniqueName: \"kubernetes.io/projected/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-kube-api-access-6x98x\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:26.808644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.808610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" event={"ID":"7c3e254a-21fb-4fd9-b6d9-502086724a94","Type":"ContainerStarted","Data":"17ec4be4fa0f879d884f74bcff2f0967ca31304863864dab653b291d501dd595"} Jul 24 22:21:26.809909 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.809887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" event={"ID":"ee29d1d3-691b-4191-aecc-54a0ae5499a7","Type":"ContainerStarted","Data":"36f6b30e98a7760e87220f1d9eaf0d76a709889338086ea974758f0a97ba0640"} Jul 24 22:21:26.851584 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.851551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:26.851584 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.851592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x98x\" (UniqueName: \"kubernetes.io/projected/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-kube-api-access-6x98x\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:26.851780 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:21:26.851703 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 22:21:26.851780 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:21:26.851768 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs podName:2d3133ce-3898-4b25-8be9-e8df2c1c23d6 nodeName:}" failed. No retries permitted until 2026-07-24 22:21:27.351751666 +0000 UTC m=+394.714190969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs") pod "model-serving-api-54d89bd5fc-mm5jq" (UID: "2d3133ce-3898-4b25-8be9-e8df2c1c23d6") : secret "model-serving-api-tls" not found Jul 24 22:21:26.879608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:26.879576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x98x\" (UniqueName: \"kubernetes.io/projected/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-kube-api-access-6x98x\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:27.118854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.118817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:27.263582 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.263499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:27.268415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.268384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:27.272675 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.272648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.276115 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.275690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 22:21:27.276115 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.275929 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 22:21:27.277097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.276498 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 22:21:27.277097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.276719 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xmv6l\"" Jul 24 22:21:27.277097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.276931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 22:21:27.285320 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.285273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:27.329276 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.329244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.355937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.355986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qlv\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.356956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.356373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:27.360172 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.360119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3133ce-3898-4b25-8be9-e8df2c1c23d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-mm5jq\" (UID: \"2d3133ce-3898-4b25-8be9-e8df2c1c23d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.458733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.457995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qlv\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.460619 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.460569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.461975 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.461946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.464126 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.464073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.465953 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.465907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.467499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.467454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.469608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.469564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qlv\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.469813 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.469789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cnsqc\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.585885 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.585827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:27.589046 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.589012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:27.765021 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.764986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mm5jq"] Jul 24 22:21:27.770818 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:27.770781 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3133ce_3898_4b25_8be9_e8df2c1c23d6.slice/crio-893ad03f7ab787c4a34ccf0c3986040dbccf424ec1ec75afbff9bfe7fe78cf97 WatchSource:0}: Error finding container 893ad03f7ab787c4a34ccf0c3986040dbccf424ec1ec75afbff9bfe7fe78cf97: Status 404 returned error can't find the container with id 893ad03f7ab787c4a34ccf0c3986040dbccf424ec1ec75afbff9bfe7fe78cf97 Jul 24 22:21:27.787062 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.787028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:27.789330 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:27.789266 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c87eba_0ee2_47b5_8e69_c3aaf9af84d1.slice/crio-c50591d1d562d45dfd7d8d8d82d357863fe352c981f758a473c07409e157027b WatchSource:0}: Error finding container c50591d1d562d45dfd7d8d8d82d357863fe352c981f758a473c07409e157027b: Status 404 returned error can't find the container with id c50591d1d562d45dfd7d8d8d82d357863fe352c981f758a473c07409e157027b Jul 24 22:21:27.819724 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.819688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" event={"ID":"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1","Type":"ContainerStarted","Data":"c50591d1d562d45dfd7d8d8d82d357863fe352c981f758a473c07409e157027b"} Jul 24 22:21:27.821271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:27.821245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" event={"ID":"2d3133ce-3898-4b25-8be9-e8df2c1c23d6","Type":"ContainerStarted","Data":"893ad03f7ab787c4a34ccf0c3986040dbccf424ec1ec75afbff9bfe7fe78cf97"} Jul 24 22:21:28.277198 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:28.277163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:31.676024 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.675982 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 22:21:31.676330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.676060 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 22:21:31.853390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.853344 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" event={"ID":"7c3e254a-21fb-4fd9-b6d9-502086724a94","Type":"ContainerStarted","Data":"2c6c83e95e3baa586e808441db40336439c35fc23d4083229c8a050ddf1cedae"} Jul 24 22:21:31.855682 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.855656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" event={"ID":"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1","Type":"ContainerStarted","Data":"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62"} Jul 24 22:21:31.855823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.855774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:31.857666 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.857602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" event={"ID":"ee29d1d3-691b-4191-aecc-54a0ae5499a7","Type":"ContainerStarted","Data":"58caa182aa114ef825b59ca65dd90ab8f32f7284601d46c4415dca16640db1e7"} Jul 24 22:21:31.857831 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.857814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:31.859396 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.859368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" event={"ID":"2d3133ce-3898-4b25-8be9-e8df2c1c23d6","Type":"ContainerStarted","Data":"5be8eb47a0a428d02f20d68c47f6cb12f91f55947bba9ac7b88723be7495cea7"} Jul 24 22:21:31.859523 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.859508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:21:31.870848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.870776 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d58b94969-q748r" podStartSLOduration=1.64013614 podStartE2EDuration="6.870762647s" totalImagesPullingTime="5.230626507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:31.869132004 +0000 UTC m=+399.231571328" watchObservedRunningTime="2026-07-24 22:21:31.870762647 +0000 UTC m=+399.233201988" Jul 24 22:21:31.885984 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.885926 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" podStartSLOduration=0.82512454 podStartE2EDuration="5.885910498s" totalImagesPullingTime="5.060785958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:31.884576713 +0000 UTC m=+399.247016036" watchObservedRunningTime="2026-07-24 22:21:31.885910498 +0000 UTC m=+399.248349821" Jul 24 22:21:31.905977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.905922 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" podStartSLOduration=2.069183559 podStartE2EDuration="5.905900649s" totalImagesPullingTime="3.83671709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:31.905440598 +0000 UTC m=+399.267879920" watchObservedRunningTime="2026-07-24 22:21:31.905900649 +0000 UTC m=+399.268339971" Jul 24 22:21:31.928355 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:31.928283 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" podStartSLOduration=1.044323664 podStartE2EDuration="4.928263731s" totalImagesPullingTime="3.883940067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:31.925409984 +0000 UTC m=+399.287849309" watchObservedRunningTime="2026-07-24 22:21:31.928263731 +0000 UTC m=+399.290703055" Jul 24 22:21:32.379742 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:32.379701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:32.438323 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:32.438249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:32.865907 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:32.865879 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cnsqc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 22:21:32.866265 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:32.865928 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 22:21:33.099648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:33.099613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:33.224344 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:33.224286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:33.324502 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:33.324471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:35.865036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:35.865010 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:36.178843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.178770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:36.253629 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.253597 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp"] Jul 24 22:21:36.257133 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.257116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.265776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.265750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 22:21:36.265905 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.265754 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qgdgl\"" Jul 24 22:21:36.278608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.278585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp"] Jul 24 22:21:36.293032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.293004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:36.342610 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.342580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmtg\" (UniqueName: \"kubernetes.io/projected/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-kube-api-access-jnmtg\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.342763 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.342642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.409982 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.409953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:36.443363 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.443340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmtg\" (UniqueName: \"kubernetes.io/projected/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-kube-api-access-jnmtg\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.443473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.443400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.446123 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.446102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.466510 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.466487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmtg\" (UniqueName: \"kubernetes.io/projected/d5a98dee-6290-4919-be2e-15fcaa8d5cc3-kube-api-access-jnmtg\") pod \"kserve-controller-manager-d6f4b65b8-9x4sp\" (UID: \"d5a98dee-6290-4919-be2e-15fcaa8d5cc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.534139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.534110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:36.568236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.568211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:36.629906 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.629873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:36.718315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.718270 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp"] Jul 24 22:21:36.720765 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:21:36.720736 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a98dee_6290_4919_be2e_15fcaa8d5cc3.slice/crio-0278d9b19742e5d6b96e7c5af3ab09fb16834fbc19badaa75f83406adfe8f084 WatchSource:0}: Error finding container 0278d9b19742e5d6b96e7c5af3ab09fb16834fbc19badaa75f83406adfe8f084: Status 404 returned error can't find the container with id 0278d9b19742e5d6b96e7c5af3ab09fb16834fbc19badaa75f83406adfe8f084 Jul 24 22:21:36.883896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:36.883867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" event={"ID":"d5a98dee-6290-4919-be2e-15fcaa8d5cc3","Type":"ContainerStarted","Data":"0278d9b19742e5d6b96e7c5af3ab09fb16834fbc19badaa75f83406adfe8f084"} Jul 24 22:21:38.784684 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:38.784644 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:38.785129 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:38.784920 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" containerID="cri-o://9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62" gracePeriod=30 Jul 24 22:21:38.864753 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:38.864718 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cnsqc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:8080/ready\": dial tcp 10.133.0.30:8080: connect: connection refused" start-of-body= Jul 24 22:21:38.864908 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:38.864775 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.30:8080/ready\": dial tcp 10.133.0.30:8080: connect: connection refused" Jul 24 22:21:39.085395 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.085376 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:39.168183 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168215 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168252 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168414 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8qlv\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168566 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token\") pod \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\" (UID: \"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1\") " Jul 24 22:21:39.168716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168691 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:21:39.168907 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.168885 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-ca-configmap\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.171052 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171012 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:21:39.171289 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171257 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts" (OuterVolumeSpecName: "cacerts") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:21:39.171444 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171420 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:21:39.171550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs" (OuterVolumeSpecName: "local-certs") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:21:39.171550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv" (OuterVolumeSpecName: "kube-api-access-n8qlv") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "kube-api-access-n8qlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:21:39.171631 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.171528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" (UID: "a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:21:39.269746 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269713 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qlv\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-kube-api-access-n8qlv\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.269746 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269741 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-token\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.269746 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269751 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-csr-dns-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.269963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269760 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-cacerts\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.269963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269768 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-istio-kubeconfig\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.269963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.269776 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1-local-certs\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:21:39.901376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.901333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" event={"ID":"d5a98dee-6290-4919-be2e-15fcaa8d5cc3","Type":"ContainerStarted","Data":"eba945fb2b097e6a199ae969eb0e8e63ea9bc573a29dc956362e4b5324e2aa95"} Jul 24 22:21:39.901732 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.901405 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:21:39.902609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.902586 2576 generic.go:356] "Generic (PLEG): container finished" podID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerID="9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62" exitCode=0 Jul 24 22:21:39.902703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.902618 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" Jul 24 22:21:39.902703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.902645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" event={"ID":"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1","Type":"ContainerDied","Data":"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62"} Jul 24 22:21:39.902703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.902678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc" event={"ID":"a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1","Type":"ContainerDied","Data":"c50591d1d562d45dfd7d8d8d82d357863fe352c981f758a473c07409e157027b"} Jul 24 22:21:39.902703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.902696 2576 scope.go:117] "RemoveContainer" containerID="9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62" Jul 24 22:21:39.911019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.911002 2576 scope.go:117] "RemoveContainer" containerID="9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62" Jul 24 22:21:39.911263 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:21:39.911245 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62\": container with ID starting with 9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62 not found: ID does not exist" containerID="9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62" Jul 24 22:21:39.911344 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.911271 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62"} err="failed to get container status \"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62\": rpc error: code = NotFound desc = could not find container \"9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62\": container with ID starting with 9585f9e60926a89581980cd2aa70040acd5718cb540dd0f2342e0b5bf138fa62 not found: ID does not exist" Jul 24 22:21:39.919723 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.919684 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" podStartSLOduration=1.6366015969999999 podStartE2EDuration="3.919670825s" totalImagesPullingTime="2.283069228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:39.918404956 +0000 UTC m=+407.280844280" watchObservedRunningTime="2026-07-24 22:21:39.919670825 +0000 UTC m=+407.282110145" Jul 24 22:21:39.934833 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.934808 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:39.938532 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:39.938513 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cnsqc"] Jul 24 22:21:41.180813 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:41.180781 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" path="/var/lib/kubelet/pods/a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1/volumes" Jul 24 22:21:41.982697 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:41.982665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:42.037146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:42.037115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:21:42.866687 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:42.866656 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-m82cl" Jul 24 22:21:42.868566 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:21:42.868544 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mm5jq" Jul 24 22:22:10.913954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:10.913924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x4sp" Jul 24 22:22:25.883648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:25.883554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:25.983513 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:25.983482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:26.762074 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.762041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:26.865699 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.865667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:26.866891 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.866869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:26.867311 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.867277 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" Jul 24 22:22:26.867401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.867314 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" Jul 24 22:22:26.867401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.867380 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c87eba-0ee2-47b5-8e69-c3aaf9af84d1" containerName="discovery" Jul 24 22:22:26.869501 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.869483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:26.872219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.872200 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 22:22:26.872366 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.872203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gdmrk\"" Jul 24 22:22:26.873278 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.873261 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 22:22:26.880652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.880628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:26.923939 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.923913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:26.965416 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:26.965393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxfp\" (UniqueName: \"kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp\") pod \"kuadrant-operator-catalog-6frs6\" (UID: \"65817dd8-2392-491b-8bef-f9c5009b5955\") " pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:27.066538 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.066472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxfp\" (UniqueName: \"kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp\") pod \"kuadrant-operator-catalog-6frs6\" (UID: \"65817dd8-2392-491b-8bef-f9c5009b5955\") " pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:27.074341 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.074284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxfp\" (UniqueName: \"kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp\") pod \"kuadrant-operator-catalog-6frs6\" (UID: \"65817dd8-2392-491b-8bef-f9c5009b5955\") " pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:27.179500 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.179476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:27.223713 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.223680 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:27.309249 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.309221 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:27.311757 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:22:27.311732 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65817dd8_2392_491b_8bef_f9c5009b5955.slice/crio-a07bb13a3975f44be408ec3ad937471431e85edbfb97e4027b9e44a6c1e80999 WatchSource:0}: Error finding container a07bb13a3975f44be408ec3ad937471431e85edbfb97e4027b9e44a6c1e80999: Status 404 returned error can't find the container with id a07bb13a3975f44be408ec3ad937471431e85edbfb97e4027b9e44a6c1e80999 Jul 24 22:22:27.431801 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.431729 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmgc4"] Jul 24 22:22:27.434666 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.434652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:27.442954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.442932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmgc4"] Jul 24 22:22:27.469250 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.469221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tg8\" (UniqueName: \"kubernetes.io/projected/7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a-kube-api-access-b6tg8\") pod \"kuadrant-operator-catalog-wmgc4\" (UID: \"7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a\") " pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:27.569710 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.569681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tg8\" (UniqueName: \"kubernetes.io/projected/7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a-kube-api-access-b6tg8\") pod \"kuadrant-operator-catalog-wmgc4\" (UID: \"7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a\") " pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:27.577431 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.577410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tg8\" (UniqueName: \"kubernetes.io/projected/7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a-kube-api-access-b6tg8\") pod \"kuadrant-operator-catalog-wmgc4\" (UID: \"7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a\") " pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:27.650930 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.650906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:27.744667 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.744634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:27.860686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.860657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:27.903204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.903179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmgc4"] Jul 24 22:22:27.905859 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:22:27.905829 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba6767e_6a37_4499_aa42_5e4e7a5eaf4a.slice/crio-b4c2106d6d28c5cd0bf402347ce8d3cd06513afe69112038055c89477dd93fd5 WatchSource:0}: Error finding container b4c2106d6d28c5cd0bf402347ce8d3cd06513afe69112038055c89477dd93fd5: Status 404 returned error can't find the container with id b4c2106d6d28c5cd0bf402347ce8d3cd06513afe69112038055c89477dd93fd5 Jul 24 22:22:27.965476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:27.965439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:28.118893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:28.118806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" event={"ID":"7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a","Type":"ContainerStarted","Data":"b4c2106d6d28c5cd0bf402347ce8d3cd06513afe69112038055c89477dd93fd5"} Jul 24 22:22:28.119962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:28.119938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" event={"ID":"65817dd8-2392-491b-8bef-f9c5009b5955","Type":"ContainerStarted","Data":"a07bb13a3975f44be408ec3ad937471431e85edbfb97e4027b9e44a6c1e80999"} Jul 24 22:22:30.131602 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.131567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" event={"ID":"7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a","Type":"ContainerStarted","Data":"49765232e93d9220f9af5b9c69357584d4010a969c9a47d423679fe0db56e65c"} Jul 24 22:22:30.133076 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.133049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" event={"ID":"65817dd8-2392-491b-8bef-f9c5009b5955","Type":"ContainerStarted","Data":"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048"} Jul 24 22:22:30.133178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.133104 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" podUID="65817dd8-2392-491b-8bef-f9c5009b5955" containerName="registry-server" containerID="cri-o://7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048" gracePeriod=2 Jul 24 22:22:30.147442 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.147399 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" podStartSLOduration=1.756084538 podStartE2EDuration="3.147383314s" totalImagesPullingTime="1.391298776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:22:30.145868734 +0000 UTC m=+457.508308055" watchObservedRunningTime="2026-07-24 22:22:30.147383314 +0000 UTC m=+457.509822635" Jul 24 22:22:30.161257 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.161217 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" podStartSLOduration=2.179372635 podStartE2EDuration="4.161201817s" totalImagesPullingTime="1.981829182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:22:30.15999866 +0000 UTC m=+457.522437983" watchObservedRunningTime="2026-07-24 22:22:30.161201817 +0000 UTC m=+457.523641138" Jul 24 22:22:30.371141 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.371116 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:30.501765 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.501736 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsxfp\" (UniqueName: \"kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp\") pod \"65817dd8-2392-491b-8bef-f9c5009b5955\" (UID: \"65817dd8-2392-491b-8bef-f9c5009b5955\") " Jul 24 22:22:30.504000 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.503968 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp" (OuterVolumeSpecName: "kube-api-access-nsxfp") pod "65817dd8-2392-491b-8bef-f9c5009b5955" (UID: "65817dd8-2392-491b-8bef-f9c5009b5955"). InnerVolumeSpecName "kube-api-access-nsxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:22:30.603189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:30.603162 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsxfp\" (UniqueName: \"kubernetes.io/projected/65817dd8-2392-491b-8bef-f9c5009b5955-kube-api-access-nsxfp\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:22:31.139088 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.139055 2576 generic.go:356] "Generic (PLEG): container finished" podID="65817dd8-2392-491b-8bef-f9c5009b5955" containerID="7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048" exitCode=0 Jul 24 22:22:31.139526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.139098 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" Jul 24 22:22:31.139526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.139132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" event={"ID":"65817dd8-2392-491b-8bef-f9c5009b5955","Type":"ContainerDied","Data":"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048"} Jul 24 22:22:31.139526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.139170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6frs6" event={"ID":"65817dd8-2392-491b-8bef-f9c5009b5955","Type":"ContainerDied","Data":"a07bb13a3975f44be408ec3ad937471431e85edbfb97e4027b9e44a6c1e80999"} Jul 24 22:22:31.139526 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.139191 2576 scope.go:117] "RemoveContainer" containerID="7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048" Jul 24 22:22:31.148307 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.148272 2576 scope.go:117] "RemoveContainer" containerID="7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048" Jul 24 22:22:31.148572 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:22:31.148553 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048\": container with ID starting with 7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048 not found: ID does not exist" containerID="7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048" Jul 24 22:22:31.148614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.148581 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048"} err="failed to get container status \"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048\": rpc error: code = NotFound desc = could not find container \"7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048\": container with ID starting with 7018e0fd5f69e33bfa76cfeeb553d0a55aebedad8ced27c93e1545f511993048 not found: ID does not exist" Jul 24 22:22:31.159045 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.159023 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:31.164359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.164337 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6frs6"] Jul 24 22:22:31.180399 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:31.180377 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65817dd8-2392-491b-8bef-f9c5009b5955" path="/var/lib/kubelet/pods/65817dd8-2392-491b-8bef-f9c5009b5955/volumes" Jul 24 22:22:37.745285 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:37.745247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:37.745742 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:37.745334 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:37.767310 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:37.767269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:38.194854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:38.194831 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wmgc4" Jul 24 22:22:50.315626 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:50.315590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:50.795206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:50.795166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:52.953413 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.953375 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9"] Jul 24 22:22:52.953940 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.953920 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65817dd8-2392-491b-8bef-f9c5009b5955" containerName="registry-server" Jul 24 22:22:52.954016 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.953942 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="65817dd8-2392-491b-8bef-f9c5009b5955" containerName="registry-server" Jul 24 22:22:52.954075 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.954060 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="65817dd8-2392-491b-8bef-f9c5009b5955" containerName="registry-server" Jul 24 22:22:52.957059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.957039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:52.960469 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.960450 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kv965\"" Jul 24 22:22:52.988839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:52.988812 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9"] Jul 24 22:22:53.091949 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.091901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2vx\" (UniqueName: \"kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" (UID: \"fb0903ad-4789-4685-817a-0d31b93a3408\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:53.192561 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.192523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2vx\" (UniqueName: \"kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" (UID: \"fb0903ad-4789-4685-817a-0d31b93a3408\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:53.202011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.201971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2vx\" (UniqueName: \"kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" (UID: \"fb0903ad-4789-4685-817a-0d31b93a3408\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:53.269754 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.269655 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kv965\"" Jul 24 22:22:53.277360 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.277334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:53.415203 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:53.415180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9"] Jul 24 22:22:53.417055 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:22:53.417020 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0903ad_4789_4685_817a_0d31b93a3408.slice/crio-cb28d23c197b42fc80234f2a5f9b71fc4a032ac9476718e97beb8a79bb677e05 WatchSource:0}: Error finding container cb28d23c197b42fc80234f2a5f9b71fc4a032ac9476718e97beb8a79bb677e05: Status 404 returned error can't find the container with id cb28d23c197b42fc80234f2a5f9b71fc4a032ac9476718e97beb8a79bb677e05 Jul 24 22:22:54.246180 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:54.246141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" event={"ID":"fb0903ad-4789-4685-817a-0d31b93a3408","Type":"ContainerStarted","Data":"cb28d23c197b42fc80234f2a5f9b71fc4a032ac9476718e97beb8a79bb677e05"} Jul 24 22:22:54.521939 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:54.521843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:55.255793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:55.255737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" event={"ID":"fb0903ad-4789-4685-817a-0d31b93a3408","Type":"ContainerStarted","Data":"4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d"} Jul 24 22:22:55.256496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:55.255856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:22:55.275096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:55.275046 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" podStartSLOduration=1.849743562 podStartE2EDuration="3.275027653s" totalImagesPullingTime="1.425284091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:22:55.273063188 +0000 UTC m=+482.635502511" watchObservedRunningTime="2026-07-24 22:22:55.275027653 +0000 UTC m=+482.637467004" Jul 24 22:22:56.621146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.621116 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5"] Jul 24 22:22:56.624658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.624642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:22:56.627933 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.627915 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 22:22:56.628484 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.628459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kdj5f\"" Jul 24 22:22:56.637945 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.637923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5"] Jul 24 22:22:56.727104 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.727072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmx8\" (UniqueName: \"kubernetes.io/projected/feb7fa2f-b93e-4f5e-822a-bb11f1c376da-kube-api-access-sbmx8\") pod \"dns-operator-controller-manager-864dd78c4c-xlpk5\" (UID: \"feb7fa2f-b93e-4f5e-822a-bb11f1c376da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:22:56.827869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.827836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmx8\" (UniqueName: \"kubernetes.io/projected/feb7fa2f-b93e-4f5e-822a-bb11f1c376da-kube-api-access-sbmx8\") pod \"dns-operator-controller-manager-864dd78c4c-xlpk5\" (UID: \"feb7fa2f-b93e-4f5e-822a-bb11f1c376da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:22:56.836710 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.836681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmx8\" (UniqueName: \"kubernetes.io/projected/feb7fa2f-b93e-4f5e-822a-bb11f1c376da-kube-api-access-sbmx8\") pod \"dns-operator-controller-manager-864dd78c4c-xlpk5\" (UID: \"feb7fa2f-b93e-4f5e-822a-bb11f1c376da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:22:56.934148 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:56.934072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:22:57.063449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:57.063419 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5"] Jul 24 22:22:57.066944 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:22:57.066918 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb7fa2f_b93e_4f5e_822a_bb11f1c376da.slice/crio-61e5eb106e790923593e91bb51f448b9e0205c96e728923430e585294ec638d6 WatchSource:0}: Error finding container 61e5eb106e790923593e91bb51f448b9e0205c96e728923430e585294ec638d6: Status 404 returned error can't find the container with id 61e5eb106e790923593e91bb51f448b9e0205c96e728923430e585294ec638d6 Jul 24 22:22:57.266392 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:57.266355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" event={"ID":"feb7fa2f-b93e-4f5e-822a-bb11f1c376da","Type":"ContainerStarted","Data":"61e5eb106e790923593e91bb51f448b9e0205c96e728923430e585294ec638d6"} Jul 24 22:22:57.322128 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:57.322091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:22:58.959691 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:22:58.959656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:00.166651 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:00.166611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:00.286591 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:00.286558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" event={"ID":"feb7fa2f-b93e-4f5e-822a-bb11f1c376da","Type":"ContainerStarted","Data":"fcd4e41b86d2ec22e2b419d299acb9d27acfe53cfe43a20bb7aa5b736433b430"} Jul 24 22:23:00.286785 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:00.286661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:23:00.326894 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:00.326848 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" podStartSLOduration=2.173392398 podStartE2EDuration="4.326830652s" totalImagesPullingTime="2.153438254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:00.325247865 +0000 UTC m=+487.687687186" watchObservedRunningTime="2026-07-24 22:23:00.326830652 +0000 UTC m=+487.689269974" Jul 24 22:23:01.235893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.235862 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr"] Jul 24 22:23:01.239283 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.239265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.241969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.241953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b4g4c\"" Jul 24 22:23:01.254920 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.254896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr"] Jul 24 22:23:01.370324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.370262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.370504 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.370428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7lr\" (UniqueName: \"kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.471795 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.471754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.471994 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.471874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7lr\" (UniqueName: \"kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.472163 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.472140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.482927 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.482894 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7lr\" (UniqueName: \"kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.549576 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.549479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:01.696061 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.695920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr"] Jul 24 22:23:01.699247 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:01.699220 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116f73c8_0f1f_4b4d_82a5_fe7bf2e3d95d.slice/crio-41d8a5993eb1cc504aaaaf0369b5fce67c1059098569a34253b7a790be573d81 WatchSource:0}: Error finding container 41d8a5993eb1cc504aaaaf0369b5fce67c1059098569a34253b7a790be573d81: Status 404 returned error can't find the container with id 41d8a5993eb1cc504aaaaf0369b5fce67c1059098569a34253b7a790be573d81 Jul 24 22:23:01.718065 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:01.718041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:02.297362 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:02.297332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" event={"ID":"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d","Type":"ContainerStarted","Data":"41d8a5993eb1cc504aaaaf0369b5fce67c1059098569a34253b7a790be573d81"} Jul 24 22:23:02.364905 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:02.364873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:02.545849 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:02.545812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:03.796994 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.794919 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c7d47cdf-w6ffk"] Jul 24 22:23:03.802108 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.802082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.807364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807174 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 22:23:03.807364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807217 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 22:23:03.807364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807282 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 22:23:03.807364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 22:23:03.807364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807174 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 22:23:03.807728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-shd2w\"" Jul 24 22:23:03.807728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807605 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 22:23:03.807728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.807722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 22:23:03.814998 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.814765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 22:23:03.820568 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.820536 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c7d47cdf-w6ffk"] Jul 24 22:23:03.879364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.879331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:03.893308 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd82x\" (UniqueName: \"kubernetes.io/projected/5eded83e-9023-44b6-ae19-d7faa9678a0d-kube-api-access-vd82x\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893441 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-oauth-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893532 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-service-ca\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-trusted-ca-bundle\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.893609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.893591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-oauth-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.975558 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.975527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:03.994816 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.994783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-service-ca\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.994960 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.994822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.994960 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.994941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-trusted-ca-bundle\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995083 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.994978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-oauth-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995083 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd82x\" (UniqueName: \"kubernetes.io/projected/5eded83e-9023-44b6-ae19-d7faa9678a0d-kube-api-access-vd82x\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995083 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-oauth-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995584 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-service-ca\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.995820 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.995804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-trusted-ca-bundle\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.996055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.996033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eded83e-9023-44b6-ae19-d7faa9678a0d-oauth-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.997520 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.997502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-oauth-config\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:03.997664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:03.997647 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eded83e-9023-44b6-ae19-d7faa9678a0d-console-serving-cert\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:04.005554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:04.005525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd82x\" (UniqueName: \"kubernetes.io/projected/5eded83e-9023-44b6-ae19-d7faa9678a0d-kube-api-access-vd82x\") pod \"console-69c7d47cdf-w6ffk\" (UID: \"5eded83e-9023-44b6-ae19-d7faa9678a0d\") " pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:04.117119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:04.117035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:04.305823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:04.305722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c7d47cdf-w6ffk"] Jul 24 22:23:04.308975 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:04.308930 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eded83e_9023_44b6_ae19_d7faa9678a0d.slice/crio-bdf136cb870821f2f2b62e4b79374eefd6bf89c7c714c6d0e3e7cf71faa29517 WatchSource:0}: Error finding container bdf136cb870821f2f2b62e4b79374eefd6bf89c7c714c6d0e3e7cf71faa29517: Status 404 returned error can't find the container with id bdf136cb870821f2f2b62e4b79374eefd6bf89c7c714c6d0e3e7cf71faa29517 Jul 24 22:23:05.316193 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:05.316157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c7d47cdf-w6ffk" event={"ID":"5eded83e-9023-44b6-ae19-d7faa9678a0d","Type":"ContainerStarted","Data":"fc295b85f90db5d58e5048b7e42a89e6ad86ca97d9153498312a59e658138ead"} Jul 24 22:23:05.316671 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:05.316209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c7d47cdf-w6ffk" event={"ID":"5eded83e-9023-44b6-ae19-d7faa9678a0d","Type":"ContainerStarted","Data":"bdf136cb870821f2f2b62e4b79374eefd6bf89c7c714c6d0e3e7cf71faa29517"} Jul 24 22:23:05.335690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:05.335644 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69c7d47cdf-w6ffk" podStartSLOduration=2.335627713 podStartE2EDuration="2.335627713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:05.333145835 +0000 UTC m=+492.695585156" watchObservedRunningTime="2026-07-24 22:23:05.335627713 +0000 UTC m=+492.698067054" Jul 24 22:23:06.130712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.130670 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb"] Jul 24 22:23:06.135045 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.135016 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:06.137539 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.137507 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nc5tl\"" Jul 24 22:23:06.143038 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.143009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb"] Jul 24 22:23:06.263871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.263838 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:06.316762 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.316726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9qx\" (UniqueName: \"kubernetes.io/projected/84081949-17e9-4332-a378-75706479ecd1-kube-api-access-jz9qx\") pod \"authorino-operator-75cc8d5cb7-bvxmb\" (UID: \"84081949-17e9-4332-a378-75706479ecd1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:06.418262 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.418169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9qx\" (UniqueName: \"kubernetes.io/projected/84081949-17e9-4332-a378-75706479ecd1-kube-api-access-jz9qx\") pod \"authorino-operator-75cc8d5cb7-bvxmb\" (UID: \"84081949-17e9-4332-a378-75706479ecd1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:06.426317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.426265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9qx\" (UniqueName: \"kubernetes.io/projected/84081949-17e9-4332-a378-75706479ecd1-kube-api-access-jz9qx\") pod \"authorino-operator-75cc8d5cb7-bvxmb\" (UID: \"84081949-17e9-4332-a378-75706479ecd1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:06.449859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.449821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:06.986376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:06.986342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb"] Jul 24 22:23:06.989131 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:06.989098 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84081949_17e9_4332_a378_75706479ecd1.slice/crio-10da7bb5a38682c6357597ebef13159f5ebaeb866f9f9520568ed3cdb68558dd WatchSource:0}: Error finding container 10da7bb5a38682c6357597ebef13159f5ebaeb866f9f9520568ed3cdb68558dd: Status 404 returned error can't find the container with id 10da7bb5a38682c6357597ebef13159f5ebaeb866f9f9520568ed3cdb68558dd Jul 24 22:23:07.327938 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:07.327900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" event={"ID":"84081949-17e9-4332-a378-75706479ecd1","Type":"ContainerStarted","Data":"10da7bb5a38682c6357597ebef13159f5ebaeb866f9f9520568ed3cdb68558dd"} Jul 24 22:23:07.329527 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:07.329493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" event={"ID":"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d","Type":"ContainerStarted","Data":"d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9"} Jul 24 22:23:07.329704 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:07.329626 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:07.353922 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:07.353877 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" podStartSLOduration=1.147202929 podStartE2EDuration="6.353861068s" totalImagesPullingTime="5.206658139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:07.35257719 +0000 UTC m=+494.715016512" watchObservedRunningTime="2026-07-24 22:23:07.353861068 +0000 UTC m=+494.716300388" Jul 24 22:23:09.121934 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.121891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:09.223113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.223067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:09.805251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.805213 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs"] Jul 24 22:23:09.808884 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.808860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.811250 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.811231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 22:23:09.811377 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.811356 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bckgb\"" Jul 24 22:23:09.811436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.811378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 22:23:09.815286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.815265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:09.816622 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.816273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs"] Jul 24 22:23:09.850217 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.850187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/348787a2-6446-4d92-a998-f3a000f2a712-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.850217 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.850217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlsl\" (UniqueName: \"kubernetes.io/projected/348787a2-6446-4d92-a998-f3a000f2a712-kube-api-access-vzlsl\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.850396 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.850235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.951397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.951368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/348787a2-6446-4d92-a998-f3a000f2a712-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.951397 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.951398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlsl\" (UniqueName: \"kubernetes.io/projected/348787a2-6446-4d92-a998-f3a000f2a712-kube-api-access-vzlsl\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.951588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.951418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.951632 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:09.951587 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 22:23:09.951668 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:09.951641 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert podName:348787a2-6446-4d92-a998-f3a000f2a712 nodeName:}" failed. No retries permitted until 2026-07-24 22:23:10.451626529 +0000 UTC m=+497.814065828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rkfhs" (UID: "348787a2-6446-4d92-a998-f3a000f2a712") : secret "plugin-serving-cert" not found Jul 24 22:23:09.952010 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.951991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/348787a2-6446-4d92-a998-f3a000f2a712-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:09.959800 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:09.959771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlsl\" (UniqueName: \"kubernetes.io/projected/348787a2-6446-4d92-a998-f3a000f2a712-kube-api-access-vzlsl\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:10.088658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.088575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:10.345708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.345615 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" event={"ID":"84081949-17e9-4332-a378-75706479ecd1","Type":"ContainerStarted","Data":"fded51bf31905a985c99ab2cc668e4f8bb13d2aa27ebcbfcc05d9791f79ac0c2"} Jul 24 22:23:10.346079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.345774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:10.365606 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.365556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" podStartSLOduration=1.984329989 podStartE2EDuration="4.36553959s" totalImagesPullingTime="2.381209601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:10.363789048 +0000 UTC m=+497.726228366" watchObservedRunningTime="2026-07-24 22:23:10.36553959 +0000 UTC m=+497.727978910" Jul 24 22:23:10.457204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.457169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:10.459609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.459584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/348787a2-6446-4d92-a998-f3a000f2a712-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rkfhs\" (UID: \"348787a2-6446-4d92-a998-f3a000f2a712\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:10.718688 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.718655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" Jul 24 22:23:10.850152 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:10.850127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs"] Jul 24 22:23:10.852347 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:10.852323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod348787a2_6446_4d92_a998_f3a000f2a712.slice/crio-346638eadd91b4dc317749f960aa38f5c21b0d5be630eae767ea5c4ca35c2a4d WatchSource:0}: Error finding container 346638eadd91b4dc317749f960aa38f5c21b0d5be630eae767ea5c4ca35c2a4d: Status 404 returned error can't find the container with id 346638eadd91b4dc317749f960aa38f5c21b0d5be630eae767ea5c4ca35c2a4d Jul 24 22:23:11.293088 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:11.293061 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xlpk5" Jul 24 22:23:11.351650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:11.351616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" event={"ID":"348787a2-6446-4d92-a998-f3a000f2a712","Type":"ContainerStarted","Data":"346638eadd91b4dc317749f960aa38f5c21b0d5be630eae767ea5c4ca35c2a4d"} Jul 24 22:23:12.130620 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:12.130587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:12.230194 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:12.230157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:14.118030 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.117983 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:14.118499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.118091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:14.123798 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.123773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:14.378153 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.377906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c7d47cdf-w6ffk" Jul 24 22:23:14.518455 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.518395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:14.574535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.574502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:14.636263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:14.636163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:16.519938 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:16.519899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:16.618851 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:16.618814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:18.337696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:18.337666 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:20.052621 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.052580 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr"] Jul 24 22:23:20.053089 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.052795 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" containerName="manager" containerID="cri-o://d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9" gracePeriod=2 Jul 24 22:23:20.072320 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.070453 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\": pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:20.078002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.077974 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr"] Jul 24 22:23:20.080868 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.080542 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:20.081663 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.081639 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" containerName="manager" Jul 24 22:23:20.081663 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.081662 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" containerName="manager" Jul 24 22:23:20.081859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.081780 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" containerName="manager" Jul 24 22:23:20.087119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.087063 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9"] Jul 24 22:23:20.087339 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.087128 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9"] Jul 24 22:23:20.087339 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.087188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.088178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.087983 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" containerName="manager" containerID="cri-o://4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d" gracePeriod=2 Jul 24 22:23:20.090345 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.090134 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:20.092679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.092652 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:20.110869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.110831 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755"] Jul 24 22:23:20.111474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.111451 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" containerName="manager" Jul 24 22:23:20.111474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.111477 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" containerName="manager" Jul 24 22:23:20.111820 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.111586 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" containerName="manager" Jul 24 22:23:20.114819 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.114783 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:20.115994 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.115974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:20.119361 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.119326 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:20.122154 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.122124 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:20.128411 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.128288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755"] Jul 24 22:23:20.149022 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:20.148977 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:20.152504 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.152452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.152673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.152623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkjm\" (UniqueName: \"kubernetes.io/projected/050e27c9-a99c-47b2-bc6f-5be844d06517-kube-api-access-wjkjm\") pod \"limitador-operator-controller-manager-859d7b78bd-nf755\" (UID: \"050e27c9-a99c-47b2-bc6f-5be844d06517\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:20.152673 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.152664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hgx\" (UniqueName: \"kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.253469 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.253429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkjm\" (UniqueName: \"kubernetes.io/projected/050e27c9-a99c-47b2-bc6f-5be844d06517-kube-api-access-wjkjm\") pod \"limitador-operator-controller-manager-859d7b78bd-nf755\" (UID: \"050e27c9-a99c-47b2-bc6f-5be844d06517\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:20.253662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.253481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hgx\" (UniqueName: \"kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.253662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.253608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.254006 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.253976 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.262922 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.262894 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkjm\" (UniqueName: \"kubernetes.io/projected/050e27c9-a99c-47b2-bc6f-5be844d06517-kube-api-access-wjkjm\") pod \"limitador-operator-controller-manager-859d7b78bd-nf755\" (UID: \"050e27c9-a99c-47b2-bc6f-5be844d06517\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:20.263060 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.262981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hgx\" (UniqueName: \"kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cd292\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.475330 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.475269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:20.483422 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:20.483388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:21.354593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:21.354561 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvxmb" Jul 24 22:23:27.956658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:27.956615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:28.137877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:28.137842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:28.236967 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:28.236879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:33.044647 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.044613 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.047039 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.047009 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.053722 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.053696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:33.065636 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.065568 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:33.068119 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.068095 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.068354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.068336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755"] Jul 24 22:23:33.070182 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.070155 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:33.133077 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.132999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:33.178625 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:33.178593 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbde35e3_bec8_4d27_9b3d_6972d479a5ee.slice/crio-2ddf6363f7a40daa18915395ac18bb7808225b47cc19c9de70946c6033a870e2 WatchSource:0}: Error finding container 2ddf6363f7a40daa18915395ac18bb7808225b47cc19c9de70946c6033a870e2: Status 404 returned error can't find the container with id 2ddf6363f7a40daa18915395ac18bb7808225b47cc19c9de70946c6033a870e2 Jul 24 22:23:33.179315 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:33.179251 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050e27c9_a99c_47b2_bc6f_5be844d06517.slice/crio-3662a7bf8b6a40cda4ad9ac905eb7cfd6683cb61fade6b7fbf5d8741a362723d WatchSource:0}: Error finding container 3662a7bf8b6a40cda4ad9ac905eb7cfd6683cb61fade6b7fbf5d8741a362723d: Status 404 returned error can't find the container with id 3662a7bf8b6a40cda4ad9ac905eb7cfd6683cb61fade6b7fbf5d8741a362723d Jul 24 22:23:33.179552 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.179523 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.181488 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.181469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh7lr\" (UniqueName: \"kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr\") pod \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " Jul 24 22:23:33.181576 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.181515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume\") pod \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\" (UID: \"116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d\") " Jul 24 22:23:33.181634 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.181597 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:33.181634 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.181624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw2vx\" (UniqueName: \"kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx\") pod \"fb0903ad-4789-4685-817a-0d31b93a3408\" (UID: \"fb0903ad-4789-4685-817a-0d31b93a3408\") " Jul 24 22:23:33.182024 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.181997 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" (UID: "116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:23:33.183630 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.183611 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr" (OuterVolumeSpecName: "kube-api-access-lh7lr") pod "116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" (UID: "116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d"). InnerVolumeSpecName "kube-api-access-lh7lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:23:33.184066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.184047 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx" (OuterVolumeSpecName: "kube-api-access-kw2vx") pod "fb0903ad-4789-4685-817a-0d31b93a3408" (UID: "fb0903ad-4789-4685-817a-0d31b93a3408"). InnerVolumeSpecName "kube-api-access-kw2vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:23:33.234558 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.234531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:33.282362 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.282336 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7lr\" (UniqueName: \"kubernetes.io/projected/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-kube-api-access-lh7lr\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:23:33.282482 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.282463 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d-extensions-socket-volume\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:23:33.282537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.282491 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2vx\" (UniqueName: \"kubernetes.io/projected/fb0903ad-4789-4685-817a-0d31b93a3408-kube-api-access-kw2vx\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:23:33.482543 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.482508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" event={"ID":"fbde35e3-bec8-4d27-9b3d-6972d479a5ee","Type":"ContainerStarted","Data":"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f"} Jul 24 22:23:33.482769 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.482557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" event={"ID":"fbde35e3-bec8-4d27-9b3d-6972d479a5ee","Type":"ContainerStarted","Data":"2ddf6363f7a40daa18915395ac18bb7808225b47cc19c9de70946c6033a870e2"} Jul 24 22:23:33.482769 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.482644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:33.484250 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.484222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" event={"ID":"050e27c9-a99c-47b2-bc6f-5be844d06517","Type":"ContainerStarted","Data":"94ceb3d510dda5a84c8a945af4430e3784833764c244eb1ae2f86f2de75477ce"} Jul 24 22:23:33.484384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.484276 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:33.484384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.484318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" event={"ID":"050e27c9-a99c-47b2-bc6f-5be844d06517","Type":"ContainerStarted","Data":"3662a7bf8b6a40cda4ad9ac905eb7cfd6683cb61fade6b7fbf5d8741a362723d"} Jul 24 22:23:33.485469 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.485448 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb0903ad-4789-4685-817a-0d31b93a3408" containerID="4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d" exitCode=0 Jul 24 22:23:33.485573 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.485480 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.485631 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.485484 2576 scope.go:117] "RemoveContainer" containerID="4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d" Jul 24 22:23:33.487096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.487075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" event={"ID":"348787a2-6446-4d92-a998-f3a000f2a712","Type":"ContainerStarted","Data":"c98f6cb258d7856069e6d92b60ca4af53fbe2382b861ba39ff793a31be926e0d"} Jul 24 22:23:33.488554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.488533 2576 generic.go:356] "Generic (PLEG): container finished" podID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" containerID="d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9" exitCode=0 Jul 24 22:23:33.488652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.488574 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:33.495055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.495036 2576 scope.go:117] "RemoveContainer" containerID="4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d" Jul 24 22:23:33.495414 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.495392 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d\": container with ID starting with 4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d not found: ID does not exist" containerID="4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d" Jul 24 22:23:33.495477 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.495425 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d"} err="failed to get container status \"4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d\": rpc error: code = NotFound desc = could not find container \"4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d\": container with ID starting with 4e0bfb9008c6389aaa808d2637bbf849fc83a5c8c06be5c4db855f6afe447c4d not found: ID does not exist" Jul 24 22:23:33.495477 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.495449 2576 scope.go:117] "RemoveContainer" containerID="d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9" Jul 24 22:23:33.504857 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.504804 2576 scope.go:117] "RemoveContainer" containerID="d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9" Jul 24 22:23:33.505183 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.505152 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9\": container with ID starting with d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9 not found: ID does not exist" containerID="d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9" Jul 24 22:23:33.505356 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.505194 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9"} err="failed to get container status \"d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9\": rpc error: code = NotFound desc = could not find container \"d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9\": container with ID starting with d34db44597dc051bdf32c56687489ae8bb752e4746e262a830fd2d9bb56a88d9 not found: ID does not exist" Jul 24 22:23:33.506912 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.506871 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" podStartSLOduration=13.506856998 podStartE2EDuration="13.506856998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:33.505761495 +0000 UTC m=+520.868200817" watchObservedRunningTime="2026-07-24 22:23:33.506856998 +0000 UTC m=+520.869296318" Jul 24 22:23:33.509542 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.509516 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zhlm9\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zhlm9" Jul 24 22:23:33.532873 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.532825 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rkfhs" podStartSLOduration=2.14072331 podStartE2EDuration="24.532808426s" totalImagesPullingTime="22.392085116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:33.531102249 +0000 UTC m=+520.893541570" watchObservedRunningTime="2026-07-24 22:23:33.532808426 +0000 UTC m=+520.895247747" Jul 24 22:23:33.533432 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:33.533405 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f4fcr\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f4fcr" Jul 24 22:23:33.556180 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:33.556127 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" podStartSLOduration=13.556109691 podStartE2EDuration="13.556109691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:33.554592814 +0000 UTC m=+520.917032136" watchObservedRunningTime="2026-07-24 22:23:33.556109691 +0000 UTC m=+520.918549011" Jul 24 22:23:35.181893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.181857 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d" path="/var/lib/kubelet/pods/116f73c8-0f1f-4b4d-82a5-fe7bf2e3d95d/volumes" Jul 24 22:23:35.182251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.182174 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0903ad-4789-4685-817a-0d31b93a3408" path="/var/lib/kubelet/pods/fb0903ad-4789-4685-817a-0d31b93a3408/volumes" Jul 24 22:23:35.257900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.257864 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:35.500174 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.500125 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" podUID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" containerName="manager" containerID="cri-o://56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f" gracePeriod=10 Jul 24 22:23:35.599421 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.599388 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:23:35.608579 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.608549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.615937 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.615911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:23:35.705192 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.705157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.705378 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.705203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zl4w\" (UniqueName: \"kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.802118 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.802084 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:35.805781 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.805755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.805926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.805811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zl4w\" (UniqueName: \"kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.806125 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.806105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.813504 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.813481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zl4w\" (UniqueName: \"kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w\") pod \"kuadrant-operator-controller-manager-86878c769f-c64gv\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:35.906774 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.906740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume\") pod \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " Jul 24 22:23:35.906958 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.906805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7hgx\" (UniqueName: \"kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx\") pod \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\" (UID: \"fbde35e3-bec8-4d27-9b3d-6972d479a5ee\") " Jul 24 22:23:35.907142 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.907119 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fbde35e3-bec8-4d27-9b3d-6972d479a5ee" (UID: "fbde35e3-bec8-4d27-9b3d-6972d479a5ee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:23:35.909053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.909035 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx" (OuterVolumeSpecName: "kube-api-access-d7hgx") pod "fbde35e3-bec8-4d27-9b3d-6972d479a5ee" (UID: "fbde35e3-bec8-4d27-9b3d-6972d479a5ee"). InnerVolumeSpecName "kube-api-access-d7hgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:23:35.920853 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:35.920831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:36.007995 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.007964 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-extensions-socket-volume\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:23:36.007995 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.007990 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hgx\" (UniqueName: \"kubernetes.io/projected/fbde35e3-bec8-4d27-9b3d-6972d479a5ee-kube-api-access-d7hgx\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:23:36.083451 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.083418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:23:36.088660 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:23:36.088630 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf8d657_5b0c_4a6f_93cf_4fad5b66300e.slice/crio-0a2b3956368abc591880274c9ad652c57c945c6fb7f189045f7af366016aa2f8 WatchSource:0}: Error finding container 0a2b3956368abc591880274c9ad652c57c945c6fb7f189045f7af366016aa2f8: Status 404 returned error can't find the container with id 0a2b3956368abc591880274c9ad652c57c945c6fb7f189045f7af366016aa2f8 Jul 24 22:23:36.134852 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.134827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:36.240663 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.240627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:36.508053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.508014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" event={"ID":"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e","Type":"ContainerStarted","Data":"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113"} Jul 24 22:23:36.508227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.508079 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:36.508227 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.508102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" event={"ID":"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e","Type":"ContainerStarted","Data":"0a2b3956368abc591880274c9ad652c57c945c6fb7f189045f7af366016aa2f8"} Jul 24 22:23:36.509364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.509340 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" containerID="56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f" exitCode=0 Jul 24 22:23:36.509483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.509372 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" Jul 24 22:23:36.509483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.509401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" event={"ID":"fbde35e3-bec8-4d27-9b3d-6972d479a5ee","Type":"ContainerDied","Data":"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f"} Jul 24 22:23:36.509483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.509447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292" event={"ID":"fbde35e3-bec8-4d27-9b3d-6972d479a5ee","Type":"ContainerDied","Data":"2ddf6363f7a40daa18915395ac18bb7808225b47cc19c9de70946c6033a870e2"} Jul 24 22:23:36.509483 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.509477 2576 scope.go:117] "RemoveContainer" containerID="56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f" Jul 24 22:23:36.519279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.519263 2576 scope.go:117] "RemoveContainer" containerID="56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f" Jul 24 22:23:36.519580 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:23:36.519562 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f\": container with ID starting with 56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f not found: ID does not exist" containerID="56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f" Jul 24 22:23:36.519623 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.519587 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f"} err="failed to get container status \"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f\": rpc error: code = NotFound desc = could not find container \"56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f\": container with ID starting with 56697dd09bc4cf9c9178489c1aca4ab1fbc2b5df23f53fd0d74c4fc95b53ed0f not found: ID does not exist" Jul 24 22:23:36.532016 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.531971 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" podStartSLOduration=1.5319579399999999 podStartE2EDuration="1.53195794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:36.528998802 +0000 UTC m=+523.891438123" watchObservedRunningTime="2026-07-24 22:23:36.53195794 +0000 UTC m=+523.894397262" Jul 24 22:23:36.544617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.544591 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:36.551062 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:36.551038 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cd292"] Jul 24 22:23:37.182488 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:37.182456 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" path="/var/lib/kubelet/pods/fbde35e3-bec8-4d27-9b3d-6972d479a5ee/volumes" Jul 24 22:23:39.864755 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:39.864719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:39.965465 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:39.965432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:44.497679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:44.497608 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nf755" Jul 24 22:23:47.518484 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:47.518452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:23:48.717472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:48.717437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:48.865507 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:48.865456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:51.310314 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:51.310261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:51.412102 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:51.412059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:51.586068 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:51.585968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:52.601565 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:52.601534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:54.510365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:54.510330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:54.615560 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:54.615529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:58.909745 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:58.909709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:23:58.965720 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:23:58.965684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:06.134624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.134577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:06.143443 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.143411 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:06.143792 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.143780 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" containerName="manager" Jul 24 22:24:06.143836 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.143794 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" containerName="manager" Jul 24 22:24:06.143892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.143883 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbde35e3-bec8-4d27-9b3d-6972d479a5ee" containerName="manager" Jul 24 22:24:06.173436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.173400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:06.173585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.173539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.176781 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.176746 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 22:24:06.253436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.253401 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:06.266591 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.266559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5hf\" (UniqueName: \"kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.266753 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.266718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.311366 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.311333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:06.367626 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.367593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5hf\" (UniqueName: \"kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.367814 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.367745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.368569 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.368546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.375442 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.375419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5hf\" (UniqueName: \"kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf\") pod \"limitador-limitador-64c94fd64f-rmnqx\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.406195 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.406125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:06.483780 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.483747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:06.621749 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.621725 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:06.623813 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:06.623782 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ece4dec_3d5b_4798_998f_70719d9085c9.slice/crio-070253de8eca20c5ba737edcce0f9aa702cdeb328526f3166d6df604fc107258 WatchSource:0}: Error finding container 070253de8eca20c5ba737edcce0f9aa702cdeb328526f3166d6df604fc107258: Status 404 returned error can't find the container with id 070253de8eca20c5ba737edcce0f9aa702cdeb328526f3166d6df604fc107258 Jul 24 22:24:06.666384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:06.666278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" event={"ID":"5ece4dec-3d5b-4798-998f-70719d9085c9","Type":"ContainerStarted","Data":"070253de8eca20c5ba737edcce0f9aa702cdeb328526f3166d6df604fc107258"} Jul 24 22:24:07.008226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.008193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:07.072390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.072356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:07.182202 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.182175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:07.186734 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.186717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:07.189272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.189246 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j8mhc\"" Jul 24 22:24:07.189923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.189903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:07.274707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.274617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtfk\" (UniqueName: \"kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk\") pod \"authorino-75d556bd77-kn77q\" (UID: \"a9c44c26-24c5-4bd5-a00b-8945af5f657f\") " pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:07.376167 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.376131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtfk\" (UniqueName: \"kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk\") pod \"authorino-75d556bd77-kn77q\" (UID: \"a9c44c26-24c5-4bd5-a00b-8945af5f657f\") " pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:07.387496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.387467 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtfk\" (UniqueName: \"kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk\") pod \"authorino-75d556bd77-kn77q\" (UID: \"a9c44c26-24c5-4bd5-a00b-8945af5f657f\") " pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:07.497248 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.497208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:07.680813 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:07.680786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:07.683534 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:07.683079 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9c44c26_24c5_4bd5_a00b_8945af5f657f.slice/crio-2a9d59afb8850a17bf1fa9f7e1be3b59b5f6ab0361a547709b67283b44fe58dd WatchSource:0}: Error finding container 2a9d59afb8850a17bf1fa9f7e1be3b59b5f6ab0361a547709b67283b44fe58dd: Status 404 returned error can't find the container with id 2a9d59afb8850a17bf1fa9f7e1be3b59b5f6ab0361a547709b67283b44fe58dd Jul 24 22:24:08.684477 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:08.684420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kn77q" event={"ID":"a9c44c26-24c5-4bd5-a00b-8945af5f657f","Type":"ContainerStarted","Data":"2a9d59afb8850a17bf1fa9f7e1be3b59b5f6ab0361a547709b67283b44fe58dd"} Jul 24 22:24:09.130383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:09.130346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:09.230674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:09.230642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:11.704709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:11.704662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kn77q" event={"ID":"a9c44c26-24c5-4bd5-a00b-8945af5f657f","Type":"ContainerStarted","Data":"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a"} Jul 24 22:24:11.706073 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:11.706048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" event={"ID":"5ece4dec-3d5b-4798-998f-70719d9085c9","Type":"ContainerStarted","Data":"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c"} Jul 24 22:24:11.706184 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:11.706168 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:11.720618 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:11.720575 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-kn77q" podStartSLOduration=1.084400638 podStartE2EDuration="4.720561658s" totalImagesPullingTime="3.63616102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:11.718738458 +0000 UTC m=+559.081177778" watchObservedRunningTime="2026-07-24 22:24:11.720561658 +0000 UTC m=+559.083000979" Jul 24 22:24:11.735277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:11.735234 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" podStartSLOduration=1.09269592 podStartE2EDuration="5.735219559s" totalImagesPullingTime="4.642523639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:11.734082587 +0000 UTC m=+559.096521908" watchObservedRunningTime="2026-07-24 22:24:11.735219559 +0000 UTC m=+559.097658908" Jul 24 22:24:22.475817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:22.475744 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:22.476409 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:22.476058 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" podUID="5ece4dec-3d5b-4798-998f-70719d9085c9" containerName="limitador" containerID="cri-o://2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c" gracePeriod=30 Jul 24 22:24:22.477094 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:22.477032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:23.028609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.028586 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:23.118594 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.118518 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file\") pod \"5ece4dec-3d5b-4798-998f-70719d9085c9\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " Jul 24 22:24:23.118594 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.118591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j5hf\" (UniqueName: \"kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf\") pod \"5ece4dec-3d5b-4798-998f-70719d9085c9\" (UID: \"5ece4dec-3d5b-4798-998f-70719d9085c9\") " Jul 24 22:24:23.118880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.118849 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file" (OuterVolumeSpecName: "config-file") pod "5ece4dec-3d5b-4798-998f-70719d9085c9" (UID: "5ece4dec-3d5b-4798-998f-70719d9085c9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:24:23.120791 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.120766 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf" (OuterVolumeSpecName: "kube-api-access-6j5hf") pod "5ece4dec-3d5b-4798-998f-70719d9085c9" (UID: "5ece4dec-3d5b-4798-998f-70719d9085c9"). InnerVolumeSpecName "kube-api-access-6j5hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:24:23.219226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.219199 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ece4dec-3d5b-4798-998f-70719d9085c9-config-file\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:23.219226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.219222 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5hf\" (UniqueName: \"kubernetes.io/projected/5ece4dec-3d5b-4798-998f-70719d9085c9-kube-api-access-6j5hf\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:23.767777 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.767744 2576 generic.go:356] "Generic (PLEG): container finished" podID="5ece4dec-3d5b-4798-998f-70719d9085c9" containerID="2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c" exitCode=0 Jul 24 22:24:23.768214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.767785 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" Jul 24 22:24:23.768214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.767830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" event={"ID":"5ece4dec-3d5b-4798-998f-70719d9085c9","Type":"ContainerDied","Data":"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c"} Jul 24 22:24:23.768214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.767880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rmnqx" event={"ID":"5ece4dec-3d5b-4798-998f-70719d9085c9","Type":"ContainerDied","Data":"070253de8eca20c5ba737edcce0f9aa702cdeb328526f3166d6df604fc107258"} Jul 24 22:24:23.768214 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.767906 2576 scope.go:117] "RemoveContainer" containerID="2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c" Jul 24 22:24:23.776674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.776660 2576 scope.go:117] "RemoveContainer" containerID="2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c" Jul 24 22:24:23.776934 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:24:23.776914 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c\": container with ID starting with 2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c not found: ID does not exist" containerID="2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c" Jul 24 22:24:23.776998 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.776950 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c"} err="failed to get container status \"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c\": rpc error: code = NotFound desc = could not find container \"2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c\": container with ID starting with 2b713b3911bd278e41d1db24e2e746a946737f37ae0152d7aa37247f60505c3c not found: ID does not exist" Jul 24 22:24:23.791096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.791064 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:23.799465 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:23.799442 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rmnqx"] Jul 24 22:24:25.181564 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:25.181531 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ece4dec-3d5b-4798-998f-70719d9085c9" path="/var/lib/kubelet/pods/5ece4dec-3d5b-4798-998f-70719d9085c9/volumes" Jul 24 22:24:27.768458 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:27.768414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:27.870577 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:27.870541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:28.230861 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.230831 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx"] Jul 24 22:24:28.231471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.231452 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ece4dec-3d5b-4798-998f-70719d9085c9" containerName="limitador" Jul 24 22:24:28.231523 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.231477 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ece4dec-3d5b-4798-998f-70719d9085c9" containerName="limitador" Jul 24 22:24:28.231596 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.231584 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ece4dec-3d5b-4798-998f-70719d9085c9" containerName="limitador" Jul 24 22:24:28.234741 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.234720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.238948 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.238928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 22:24:28.238948 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.238938 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qngfp\"" Jul 24 22:24:28.240050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.240028 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 22:24:28.240178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.240075 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 22:24:28.242895 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.242826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx"] Jul 24 22:24:28.278470 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.278435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:28.368674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.368643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee7e87-0514-43ba-8008-252a9a8c7d15-data\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.368674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.368678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qhz\" (UniqueName: \"kubernetes.io/projected/e5ee7e87-0514-43ba-8008-252a9a8c7d15-kube-api-access-p6qhz\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.469621 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.469587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee7e87-0514-43ba-8008-252a9a8c7d15-data\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.469621 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.469624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qhz\" (UniqueName: \"kubernetes.io/projected/e5ee7e87-0514-43ba-8008-252a9a8c7d15-kube-api-access-p6qhz\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.469951 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.469930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee7e87-0514-43ba-8008-252a9a8c7d15-data\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.478182 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.478154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qhz\" (UniqueName: \"kubernetes.io/projected/e5ee7e87-0514-43ba-8008-252a9a8c7d15-kube-api-access-p6qhz\") pod \"postgres-69fb4594cf-vbbqx\" (UID: \"e5ee7e87-0514-43ba-8008-252a9a8c7d15\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.546897 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.546832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:28.678707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.678547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx"] Jul 24 22:24:28.681687 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:28.681651 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ee7e87_0514_43ba_8008_252a9a8c7d15.slice/crio-184e099d63e3997e1bfdf0b579d0046eccb9cd7f6c4a56eba37158e0e48e9b45 WatchSource:0}: Error finding container 184e099d63e3997e1bfdf0b579d0046eccb9cd7f6c4a56eba37158e0e48e9b45: Status 404 returned error can't find the container with id 184e099d63e3997e1bfdf0b579d0046eccb9cd7f6c4a56eba37158e0e48e9b45 Jul 24 22:24:28.804686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:28.804596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" event={"ID":"e5ee7e87-0514-43ba-8008-252a9a8c7d15","Type":"ContainerStarted","Data":"184e099d63e3997e1bfdf0b579d0046eccb9cd7f6c4a56eba37158e0e48e9b45"} Jul 24 22:24:33.835824 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:33.835793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" event={"ID":"e5ee7e87-0514-43ba-8008-252a9a8c7d15","Type":"ContainerStarted","Data":"2d5e068d7ffd3c7787ddf78a7435eb170ca90c919154712c2dbe20358a0d30d3"} Jul 24 22:24:33.836207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:33.835916 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:33.852200 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:33.852156 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" podStartSLOduration=0.770686956 podStartE2EDuration="5.852140577s" totalImagesPullingTime="5.081453621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:33.850568239 +0000 UTC m=+581.213007561" watchObservedRunningTime="2026-07-24 22:24:33.852140577 +0000 UTC m=+581.214579897" Jul 24 22:24:39.869433 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:39.869403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vbbqx" Jul 24 22:24:40.174349 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.174247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:40.274686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.274650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:40.386501 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.386452 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wkmwv"] Jul 24 22:24:40.393243 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.393064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.395461 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.395439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wkmwv"] Jul 24 22:24:40.479788 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.479756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpls4\" (UniqueName: \"kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4\") pod \"authorino-5d4b67c446-wkmwv\" (UID: \"0d5836d4-6b5a-4424-97cf-67498dfaffca\") " pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.580484 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.580454 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wkmwv"] Jul 24 22:24:40.580713 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.580681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpls4\" (UniqueName: \"kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4\") pod \"authorino-5d4b67c446-wkmwv\" (UID: \"0d5836d4-6b5a-4424-97cf-67498dfaffca\") " pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.580944 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:24:40.580800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tpls4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-wkmwv" podUID="0d5836d4-6b5a-4424-97cf-67498dfaffca" Jul 24 22:24:40.591789 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.591764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpls4\" (UniqueName: \"kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4\") pod \"authorino-5d4b67c446-wkmwv\" (UID: \"0d5836d4-6b5a-4424-97cf-67498dfaffca\") " pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.607279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.607254 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-m5z27"] Jul 24 22:24:40.611484 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.611467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.613947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.613928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 22:24:40.617064 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.617042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-m5z27"] Jul 24 22:24:40.646516 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.646494 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-m5z27"] Jul 24 22:24:40.646750 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:24:40.646733 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5kdsq tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-5kdsq tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-m5z27" podUID="ef933424-18d2-46d3-a19b-5341df01f159" Jul 24 22:24:40.680810 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.680783 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:24:40.682329 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.682285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdsq\" (UniqueName: \"kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.682420 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.682341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.684589 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.684572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.691158 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.691140 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:24:40.783536 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.783453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdsq\" (UniqueName: \"kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.783536 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.783507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.783757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.783572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc4n\" (UniqueName: \"kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.783757 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.783616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.786143 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.786120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.795770 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.795748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdsq\" (UniqueName: \"kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq\") pod \"authorino-76877994-m5z27\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.869246 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.869221 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.869385 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.869222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.876859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.876841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:40.879921 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.879904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:40.884071 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.884055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc4n\" (UniqueName: \"kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.884131 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.884090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.886439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.886416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.906701 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.906678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc4n\" (UniqueName: \"kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n\") pod \"authorino-85bc4bbc76-zdnkk\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:40.984975 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.984950 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpls4\" (UniqueName: \"kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4\") pod \"0d5836d4-6b5a-4424-97cf-67498dfaffca\" (UID: \"0d5836d4-6b5a-4424-97cf-67498dfaffca\") " Jul 24 22:24:40.985121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.984992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kdsq\" (UniqueName: \"kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq\") pod \"ef933424-18d2-46d3-a19b-5341df01f159\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " Jul 24 22:24:40.985121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.985062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert\") pod \"ef933424-18d2-46d3-a19b-5341df01f159\" (UID: \"ef933424-18d2-46d3-a19b-5341df01f159\") " Jul 24 22:24:40.987205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.987182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ef933424-18d2-46d3-a19b-5341df01f159" (UID: "ef933424-18d2-46d3-a19b-5341df01f159"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:24:40.987205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.987187 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq" (OuterVolumeSpecName: "kube-api-access-5kdsq") pod "ef933424-18d2-46d3-a19b-5341df01f159" (UID: "ef933424-18d2-46d3-a19b-5341df01f159"). InnerVolumeSpecName "kube-api-access-5kdsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:24:40.987375 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.987203 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4" (OuterVolumeSpecName: "kube-api-access-tpls4") pod "0d5836d4-6b5a-4424-97cf-67498dfaffca" (UID: "0d5836d4-6b5a-4424-97cf-67498dfaffca"). InnerVolumeSpecName "kube-api-access-tpls4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:24:40.994460 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:40.994445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:24:41.086057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.086024 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdsq\" (UniqueName: \"kubernetes.io/projected/ef933424-18d2-46d3-a19b-5341df01f159-kube-api-access-5kdsq\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:41.086176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.086063 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef933424-18d2-46d3-a19b-5341df01f159-tls-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:41.086176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.086081 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpls4\" (UniqueName: \"kubernetes.io/projected/0d5836d4-6b5a-4424-97cf-67498dfaffca-kube-api-access-tpls4\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:41.126769 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.126745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:24:41.128982 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:41.128942 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a1e8ff_7ad4_4ea5_883c_1a1af5733a66.slice/crio-01c073ca00bb0e5a47a592140bc5db8e01badb237506d3f95392854afc63d346 WatchSource:0}: Error finding container 01c073ca00bb0e5a47a592140bc5db8e01badb237506d3f95392854afc63d346: Status 404 returned error can't find the container with id 01c073ca00bb0e5a47a592140bc5db8e01badb237506d3f95392854afc63d346 Jul 24 22:24:41.875597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.875556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" event={"ID":"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66","Type":"ContainerStarted","Data":"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5"} Jul 24 22:24:41.875597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.875599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wkmwv" Jul 24 22:24:41.875844 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.875610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" event={"ID":"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66","Type":"ContainerStarted","Data":"01c073ca00bb0e5a47a592140bc5db8e01badb237506d3f95392854afc63d346"} Jul 24 22:24:41.875844 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.875610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-m5z27" Jul 24 22:24:41.896743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.896696 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" podStartSLOduration=1.5922269 podStartE2EDuration="1.896681922s" totalImagesPullingTime="304.455022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:41.894195186 +0000 UTC m=+589.256634506" watchObservedRunningTime="2026-07-24 22:24:41.896681922 +0000 UTC m=+589.259121242" Jul 24 22:24:41.924355 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.924322 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wkmwv"] Jul 24 22:24:41.926339 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.926311 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:41.926511 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.926489 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-kn77q" podUID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" containerName="authorino" containerID="cri-o://8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a" gracePeriod=30 Jul 24 22:24:41.928144 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.928121 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wkmwv"] Jul 24 22:24:41.953782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.953756 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-m5z27"] Jul 24 22:24:41.956793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:41.956765 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-m5z27"] Jul 24 22:24:42.190088 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.190066 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:42.296266 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.296238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rtfk\" (UniqueName: \"kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk\") pod \"a9c44c26-24c5-4bd5-a00b-8945af5f657f\" (UID: \"a9c44c26-24c5-4bd5-a00b-8945af5f657f\") " Jul 24 22:24:42.298418 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.298388 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk" (OuterVolumeSpecName: "kube-api-access-4rtfk") pod "a9c44c26-24c5-4bd5-a00b-8945af5f657f" (UID: "a9c44c26-24c5-4bd5-a00b-8945af5f657f"). InnerVolumeSpecName "kube-api-access-4rtfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:24:42.397046 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.397013 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtfk\" (UniqueName: \"kubernetes.io/projected/a9c44c26-24c5-4bd5-a00b-8945af5f657f-kube-api-access-4rtfk\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:24:42.872134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.872101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:42.882432 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.882407 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" containerID="8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a" exitCode=0 Jul 24 22:24:42.882570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.882456 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kn77q" Jul 24 22:24:42.882570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.882479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kn77q" event={"ID":"a9c44c26-24c5-4bd5-a00b-8945af5f657f","Type":"ContainerDied","Data":"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a"} Jul 24 22:24:42.882570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.882514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kn77q" event={"ID":"a9c44c26-24c5-4bd5-a00b-8945af5f657f","Type":"ContainerDied","Data":"2a9d59afb8850a17bf1fa9f7e1be3b59b5f6ab0361a547709b67283b44fe58dd"} Jul 24 22:24:42.882570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.882526 2576 scope.go:117] "RemoveContainer" containerID="8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a" Jul 24 22:24:42.891376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.891360 2576 scope.go:117] "RemoveContainer" containerID="8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a" Jul 24 22:24:42.891625 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:24:42.891608 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a\": container with ID starting with 8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a not found: ID does not exist" containerID="8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a" Jul 24 22:24:42.891675 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.891633 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a"} err="failed to get container status \"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a\": rpc error: code = NotFound desc = could not find container \"8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a\": container with ID starting with 8c911bb9bcf2d8675fb8cd146319c553794aceaa52a93d3a7f2b8d6b11fc300a not found: ID does not exist" Jul 24 22:24:42.904026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.904004 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:42.905795 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:42.905778 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kn77q"] Jul 24 22:24:43.183790 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:43.183705 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5836d4-6b5a-4424-97cf-67498dfaffca" path="/var/lib/kubelet/pods/0d5836d4-6b5a-4424-97cf-67498dfaffca/volumes" Jul 24 22:24:43.184070 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:43.184047 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" path="/var/lib/kubelet/pods/a9c44c26-24c5-4bd5-a00b-8945af5f657f/volumes" Jul 24 22:24:43.184573 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:43.184549 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef933424-18d2-46d3-a19b-5341df01f159" path="/var/lib/kubelet/pods/ef933424-18d2-46d3-a19b-5341df01f159/volumes" Jul 24 22:24:46.457331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.457288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:46.476658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.476625 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:24:46.477085 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.477062 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" containerName="authorino" Jul 24 22:24:46.477085 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.477084 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" containerName="authorino" Jul 24 22:24:46.477321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.477202 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c44c26-24c5-4bd5-a00b-8945af5f657f" containerName="authorino" Jul 24 22:24:46.553168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.553136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:46.569590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.569566 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:24:46.569750 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.569670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.573871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.573848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64zqd\"" Jul 24 22:24:46.573871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.573871 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 22:24:46.574029 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.573873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 22:24:46.627107 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.627075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:46.632372 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.632345 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:24:46.635633 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.635608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.635726 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.635642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2mm\" (UniqueName: \"kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.672364 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.672337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:24:46.672514 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.672389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.736719 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.736609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.736719 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.736642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2mm\" (UniqueName: \"kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.736719 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.736706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.736951 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.736749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvtp\" (UniqueName: \"kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.739151 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.739123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.749202 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.749176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2mm\" (UniqueName: \"kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm\") pod \"maas-controller-766f4864db-2wmqk\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.837539 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.837510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.837645 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.837557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvtp\" (UniqueName: \"kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.839973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.839956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.846947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.846920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvtp\" (UniqueName: \"kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp\") pod \"maas-controller-656df4d8c4-fcwwr\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:46.880588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.880544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:46.982144 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:46.982117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:47.043219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:47.043187 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:24:47.045821 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:47.045796 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cd5751_f15c_46ff_a362_7265412366ff.slice/crio-6051d00f5069fc5675aefa25818db13c9befdd26320338b81340f6fa6953bab1 WatchSource:0}: Error finding container 6051d00f5069fc5675aefa25818db13c9befdd26320338b81340f6fa6953bab1: Status 404 returned error can't find the container with id 6051d00f5069fc5675aefa25818db13c9befdd26320338b81340f6fa6953bab1 Jul 24 22:24:47.150368 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:47.150346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:24:47.152183 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:24:47.152159 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa294a92_0036_48fe_a77a_a5b455a482f5.slice/crio-14bf338d509b55d758f7797c15abcbe4ec030cb37a64877edb0b5736aaa033ab WatchSource:0}: Error finding container 14bf338d509b55d758f7797c15abcbe4ec030cb37a64877edb0b5736aaa033ab: Status 404 returned error can't find the container with id 14bf338d509b55d758f7797c15abcbe4ec030cb37a64877edb0b5736aaa033ab Jul 24 22:24:47.916700 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:47.916654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" event={"ID":"aa294a92-0036-48fe-a77a-a5b455a482f5","Type":"ContainerStarted","Data":"14bf338d509b55d758f7797c15abcbe4ec030cb37a64877edb0b5736aaa033ab"} Jul 24 22:24:47.921447 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:47.921412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766f4864db-2wmqk" event={"ID":"68cd5751-f15c-46ff-a362-7265412366ff","Type":"ContainerStarted","Data":"6051d00f5069fc5675aefa25818db13c9befdd26320338b81340f6fa6953bab1"} Jul 24 22:24:49.935220 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.935184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:49.936759 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.936725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766f4864db-2wmqk" event={"ID":"68cd5751-f15c-46ff-a362-7265412366ff","Type":"ContainerStarted","Data":"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8"} Jul 24 22:24:49.936929 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.936911 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:24:49.938585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.938558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" event={"ID":"aa294a92-0036-48fe-a77a-a5b455a482f5","Type":"ContainerStarted","Data":"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8"} Jul 24 22:24:49.938708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.938613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:24:49.958770 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.958723 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766f4864db-2wmqk" podStartSLOduration=1.269107174 podStartE2EDuration="3.958708096s" totalImagesPullingTime="2.689600922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:49.957749835 +0000 UTC m=+597.320189157" watchObservedRunningTime="2026-07-24 22:24:49.958708096 +0000 UTC m=+597.321147458" Jul 24 22:24:49.980470 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:49.980424 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" podStartSLOduration=1.387507461 podStartE2EDuration="3.980409846s" totalImagesPullingTime="2.592902385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:49.977921207 +0000 UTC m=+597.340360532" watchObservedRunningTime="2026-07-24 22:24:49.980409846 +0000 UTC m=+597.342849166" Jul 24 22:24:50.023764 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:50.023676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:24:53.055111 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:53.055085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:24:53.055682 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:53.055418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:24:53.062642 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:53.062622 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:24:53.062793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:53.062679 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:24:57.513116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:24:57.513041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:00.945942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:00.945911 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:25:00.947500 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:00.945982 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:25:01.011265 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.011231 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:25:01.011571 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.011468 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766f4864db-2wmqk" podUID="68cd5751-f15c-46ff-a362-7265412366ff" containerName="manager" containerID="cri-o://a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8" gracePeriod=30 Jul 24 22:25:01.265979 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.265957 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:25:01.372609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.372575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c2mm\" (UniqueName: \"kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm\") pod \"68cd5751-f15c-46ff-a362-7265412366ff\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " Jul 24 22:25:01.372783 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.372677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert\") pod \"68cd5751-f15c-46ff-a362-7265412366ff\" (UID: \"68cd5751-f15c-46ff-a362-7265412366ff\") " Jul 24 22:25:01.374872 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.374847 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68cd5751-f15c-46ff-a362-7265412366ff" (UID: "68cd5751-f15c-46ff-a362-7265412366ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:25:01.374979 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.374902 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm" (OuterVolumeSpecName: "kube-api-access-5c2mm") pod "68cd5751-f15c-46ff-a362-7265412366ff" (UID: "68cd5751-f15c-46ff-a362-7265412366ff"). InnerVolumeSpecName "kube-api-access-5c2mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:25:01.474001 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.473969 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c2mm\" (UniqueName: \"kubernetes.io/projected/68cd5751-f15c-46ff-a362-7265412366ff-kube-api-access-5c2mm\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:25:01.474001 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:01.473997 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cd5751-f15c-46ff-a362-7265412366ff-webhook-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:25:02.005549 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.005512 2576 generic.go:356] "Generic (PLEG): container finished" podID="68cd5751-f15c-46ff-a362-7265412366ff" containerID="a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8" exitCode=0 Jul 24 22:25:02.005549 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.005555 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766f4864db-2wmqk" Jul 24 22:25:02.006066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.005605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766f4864db-2wmqk" event={"ID":"68cd5751-f15c-46ff-a362-7265412366ff","Type":"ContainerDied","Data":"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8"} Jul 24 22:25:02.006066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.005667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766f4864db-2wmqk" event={"ID":"68cd5751-f15c-46ff-a362-7265412366ff","Type":"ContainerDied","Data":"6051d00f5069fc5675aefa25818db13c9befdd26320338b81340f6fa6953bab1"} Jul 24 22:25:02.006066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.005703 2576 scope.go:117] "RemoveContainer" containerID="a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8" Jul 24 22:25:02.015386 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.015183 2576 scope.go:117] "RemoveContainer" containerID="a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8" Jul 24 22:25:02.015572 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:25:02.015554 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8\": container with ID starting with a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8 not found: ID does not exist" containerID="a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8" Jul 24 22:25:02.015621 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.015579 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8"} err="failed to get container status \"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8\": rpc error: code = NotFound desc = could not find container \"a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8\": container with ID starting with a796bff2db15a425335dc5e6f6571cc641530cc2ebf55f0dd1b2b75d168046f8 not found: ID does not exist" Jul 24 22:25:02.027174 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.027150 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:25:02.030847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:02.030827 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766f4864db-2wmqk"] Jul 24 22:25:03.182859 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:03.182831 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68cd5751-f15c-46ff-a362-7265412366ff" path="/var/lib/kubelet/pods/68cd5751-f15c-46ff-a362-7265412366ff/volumes" Jul 24 22:25:16.588843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:16.588750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:20.221689 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.221659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:20.323644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.323603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:20.702116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.702084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:20.847351 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.847319 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl"] Jul 24 22:25:20.847729 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.847708 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cd5751-f15c-46ff-a362-7265412366ff" containerName="manager" Jul 24 22:25:20.847729 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.847723 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cd5751-f15c-46ff-a362-7265412366ff" containerName="manager" Jul 24 22:25:20.847973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.847790 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cd5751-f15c-46ff-a362-7265412366ff" containerName="manager" Jul 24 22:25:20.852468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.852450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:20.854981 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.854959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 22:25:20.862922 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.862898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl"] Jul 24 22:25:20.925591 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.925564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:20.951157 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.951130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfqw\" (UniqueName: \"kubernetes.io/projected/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-kube-api-access-pvfqw\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:20.951352 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.951189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-openshift-service-ca\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:20.951352 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:20.951241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-maas-api-tls\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.051692 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.051605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-maas-api-tls\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.051692 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.051654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfqw\" (UniqueName: \"kubernetes.io/projected/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-kube-api-access-pvfqw\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.051908 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.051706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-openshift-service-ca\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.052263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.052243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-openshift-service-ca\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.054139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.054115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-maas-api-tls\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.060432 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.060410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfqw\" (UniqueName: \"kubernetes.io/projected/78c7fee3-f730-42f3-ba04-8ac28c6b1ee7-kube-api-access-pvfqw\") pod \"maas-api-79bc7645cc-txtvl\" (UID: \"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7\") " pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.164555 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.164520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:21.267816 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.267746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:21.314415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.314383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl"] Jul 24 22:25:21.316323 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:25:21.316276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c7fee3_f730_42f3_ba04_8ac28c6b1ee7.slice/crio-8bb302566fe21c805a8835ac6a7d4ce9bfff5caf04c3a6b73217f1ee1eb1f55b WatchSource:0}: Error finding container 8bb302566fe21c805a8835ac6a7d4ce9bfff5caf04c3a6b73217f1ee1eb1f55b: Status 404 returned error can't find the container with id 8bb302566fe21c805a8835ac6a7d4ce9bfff5caf04c3a6b73217f1ee1eb1f55b Jul 24 22:25:21.377885 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.377845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:21.387648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.387617 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl"] Jul 24 22:25:21.390729 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.390715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.393533 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.393511 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8l4rx\"" Jul 24 22:25:21.393646 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.393514 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 22:25:21.402562 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.402540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl"] Jul 24 22:25:21.455837 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.455803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad42bcf7-ee9a-440d-9abe-42a052d59018-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.456003 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.455918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnsz\" (UniqueName: \"kubernetes.io/projected/ad42bcf7-ee9a-440d-9abe-42a052d59018-kube-api-access-qrnsz\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.556839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.556810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnsz\" (UniqueName: \"kubernetes.io/projected/ad42bcf7-ee9a-440d-9abe-42a052d59018-kube-api-access-qrnsz\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.556996 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.556892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad42bcf7-ee9a-440d-9abe-42a052d59018-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.557585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.557565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad42bcf7-ee9a-440d-9abe-42a052d59018-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.572690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.572621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnsz\" (UniqueName: \"kubernetes.io/projected/ad42bcf7-ee9a-440d-9abe-42a052d59018-kube-api-access-qrnsz\") pod \"payload-pre-processing-687dbb6cf6-49gcl\" (UID: \"ad42bcf7-ee9a-440d-9abe-42a052d59018\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.702788 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.702749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:21.841062 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:21.840979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl"] Jul 24 22:25:21.844859 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:25:21.844831 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad42bcf7_ee9a_440d_9abe_42a052d59018.slice/crio-962ea8e62e5625a2cc88f7f5ca5d4efeefb5081a75b9e33b2827fc9c9847d036 WatchSource:0}: Error finding container 962ea8e62e5625a2cc88f7f5ca5d4efeefb5081a75b9e33b2827fc9c9847d036: Status 404 returned error can't find the container with id 962ea8e62e5625a2cc88f7f5ca5d4efeefb5081a75b9e33b2827fc9c9847d036 Jul 24 22:25:22.114747 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:22.114659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" event={"ID":"ad42bcf7-ee9a-440d-9abe-42a052d59018","Type":"ContainerStarted","Data":"962ea8e62e5625a2cc88f7f5ca5d4efeefb5081a75b9e33b2827fc9c9847d036"} Jul 24 22:25:22.116762 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:22.116732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" event={"ID":"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7","Type":"ContainerStarted","Data":"8bb302566fe21c805a8835ac6a7d4ce9bfff5caf04c3a6b73217f1ee1eb1f55b"} Jul 24 22:25:23.126179 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:23.126136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" event={"ID":"78c7fee3-f730-42f3-ba04-8ac28c6b1ee7","Type":"ContainerStarted","Data":"ab924d5847e85eeab375559963b7306b6390a1331a4b16bb26089096a355eb38"} Jul 24 22:25:23.126656 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:23.126338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:23.145025 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:23.144979 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" podStartSLOduration=1.5360059449999999 podStartE2EDuration="3.144961769s" totalImagesPullingTime="1.608955824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:23.142339535 +0000 UTC m=+630.504778857" watchObservedRunningTime="2026-07-24 22:25:23.144961769 +0000 UTC m=+630.507401092" Jul 24 22:25:24.133466 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:24.133421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" event={"ID":"ad42bcf7-ee9a-440d-9abe-42a052d59018","Type":"ContainerStarted","Data":"ddf44f926a9479f64f8dd5055216477cec4f9971760f8dfafaac26550548d33f"} Jul 24 22:25:24.133971 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:24.133507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:24.151041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:24.150990 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" podStartSLOduration=1.4571012429999999 podStartE2EDuration="3.150975726s" totalImagesPullingTime="1.693874483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:24.149248512 +0000 UTC m=+631.511687835" watchObservedRunningTime="2026-07-24 22:25:24.150975726 +0000 UTC m=+631.513415063" Jul 24 22:25:29.139927 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:29.139892 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79bc7645cc-txtvl" Jul 24 22:25:30.116053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:30.116021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:31.473187 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:31.473150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:35.140245 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:35.140214 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-49gcl" Jul 24 22:25:36.979556 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:36.979518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:43.390834 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.390803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c"] Jul 24 22:25:43.394162 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.394145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.397984 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.397956 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 22:25:43.397984 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.397977 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 22:25:43.398185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.397961 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 22:25:43.398185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.397968 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v4bdf\"" Jul 24 22:25:43.404819 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.404793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c"] Jul 24 22:25:43.439774 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.439739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:43.457979 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.457947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.458085 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.457982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.458085 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.458020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.458169 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.458080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsltx\" (UniqueName: \"kubernetes.io/projected/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-kube-api-access-fsltx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.458169 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.458125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.458169 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.458156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559387 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsltx\" (UniqueName: \"kubernetes.io/projected/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-kube-api-access-fsltx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559850 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559850 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.559965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.559868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.561973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.561943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.562136 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.562120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.567553 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.567532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsltx\" (UniqueName: \"kubernetes.io/projected/f9778b4e-e7f3-4f7a-a257-644ee9ee9897-kube-api-access-fsltx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qml8c\" (UID: \"f9778b4e-e7f3-4f7a-a257-644ee9ee9897\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.706024 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.705997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:43.844617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.844593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c"] Jul 24 22:25:43.846588 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:25:43.846564 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9778b4e_e7f3_4f7a_a257_644ee9ee9897.slice/crio-7e1590703ca79884bf38e845d6961975901bd7e810f6cc470c3c3f17e1216638 WatchSource:0}: Error finding container 7e1590703ca79884bf38e845d6961975901bd7e810f6cc470c3c3f17e1216638: Status 404 returned error can't find the container with id 7e1590703ca79884bf38e845d6961975901bd7e810f6cc470c3c3f17e1216638 Jul 24 22:25:43.848259 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:43.848240 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:25:44.239742 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:44.239703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" event={"ID":"f9778b4e-e7f3-4f7a-a257-644ee9ee9897","Type":"ContainerStarted","Data":"7e1590703ca79884bf38e845d6961975901bd7e810f6cc470c3c3f17e1216638"} Jul 24 22:25:44.515355 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:44.515243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:44.612451 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:44.612420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:45.245623 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:45.245591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" event={"ID":"f9778b4e-e7f3-4f7a-a257-644ee9ee9897","Type":"ContainerStarted","Data":"229354750f41f8103ebd4974ea4b268a08b64b3e7a0e3bbe5c7788c352a774b9"} Jul 24 22:25:45.264328 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:45.264269 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" podStartSLOduration=1.257962525 podStartE2EDuration="2.264253829s" totalImagesPullingTime="1.006291304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:45.261670654 +0000 UTC m=+652.624109976" watchObservedRunningTime="2026-07-24 22:25:45.264253829 +0000 UTC m=+652.626693150" Jul 24 22:25:50.420365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:50.420327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:25:53.706211 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:53.706173 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:53.706742 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:53.706491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:53.719014 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:53.718992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:54.303119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:54.303090 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qml8c" Jul 24 22:25:56.271022 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:25:56.270992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:26:12.214360 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.214330 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx"] Jul 24 22:26:12.218375 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.218354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.221153 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.221135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 22:26:12.232018 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.231989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx"] Jul 24 22:26:12.244105 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.244078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:26:12.327228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.327228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.327453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.327453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42wh\" (UniqueName: \"kubernetes.io/projected/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-kube-api-access-k42wh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.327453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.327453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.327427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428261 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428261 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428261 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428430 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42wh\" (UniqueName: \"kubernetes.io/projected/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-kube-api-access-k42wh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428480 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428804 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.428884 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.428837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.430433 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.430415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.430716 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.430699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.437153 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.437131 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42wh\" (UniqueName: \"kubernetes.io/projected/d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8-kube-api-access-k42wh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx\" (UID: \"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.528166 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.528085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:12.662538 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:12.662512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx"] Jul 24 22:26:12.664392 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:26:12.664362 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c7c1a6_d3e7_48cd_8b15_5bc565a954a8.slice/crio-8d37f7edbdbf6b7228de47ef924992e7f2fb188a434eeb9c75d82cb2ee46a508 WatchSource:0}: Error finding container 8d37f7edbdbf6b7228de47ef924992e7f2fb188a434eeb9c75d82cb2ee46a508: Status 404 returned error can't find the container with id 8d37f7edbdbf6b7228de47ef924992e7f2fb188a434eeb9c75d82cb2ee46a508 Jul 24 22:26:13.390316 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:13.390249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" event={"ID":"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8","Type":"ContainerStarted","Data":"d1de7854e0b5e90dc5cb641344e272fb575e3c1eeec1ec3083aa7a278dc813b0"} Jul 24 22:26:13.390756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:13.390373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" event={"ID":"d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8","Type":"ContainerStarted","Data":"8d37f7edbdbf6b7228de47ef924992e7f2fb188a434eeb9c75d82cb2ee46a508"} Jul 24 22:26:13.409993 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:13.409945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" podStartSLOduration=1.234818564 podStartE2EDuration="1.409930476s" totalImagesPullingTime="175.111912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:26:13.40826315 +0000 UTC m=+680.770702472" watchObservedRunningTime="2026-07-24 22:26:13.409930476 +0000 UTC m=+680.772369850" Jul 24 22:26:18.500020 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:18.499978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:26:22.528259 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:22.528224 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:22.528664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:22.528285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:22.541039 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:22.541009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:23.454244 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:23.454216 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx" Jul 24 22:26:30.728402 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:30.728368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:26:34.539779 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:34.539730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:26:50.012473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:26:50.012368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:27:59.138365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.138325 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f6b445f89-kjqvv"] Jul 24 22:27:59.141578 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.141562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.149677 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.149651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6b445f89-kjqvv"] Jul 24 22:27:59.222155 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.222116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3953506-d593-4f75-8458-6832d4cfd8bc-tls-cert\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.222373 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.222258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96dg\" (UniqueName: \"kubernetes.io/projected/c3953506-d593-4f75-8458-6832d4cfd8bc-kube-api-access-l96dg\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.323431 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.323389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96dg\" (UniqueName: \"kubernetes.io/projected/c3953506-d593-4f75-8458-6832d4cfd8bc-kube-api-access-l96dg\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.323617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.323455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3953506-d593-4f75-8458-6832d4cfd8bc-tls-cert\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.326056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.326028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3953506-d593-4f75-8458-6832d4cfd8bc-tls-cert\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.331383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.331357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96dg\" (UniqueName: \"kubernetes.io/projected/c3953506-d593-4f75-8458-6832d4cfd8bc-kube-api-access-l96dg\") pod \"authorino-f6b445f89-kjqvv\" (UID: \"c3953506-d593-4f75-8458-6832d4cfd8bc\") " pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.447977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.447936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:27:59.451824 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.451790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6b445f89-kjqvv" Jul 24 22:27:59.547185 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.547159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:27:59.586936 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.586913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6b445f89-kjqvv"] Jul 24 22:27:59.589772 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:27:59.589744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3953506_d593_4f75_8458_6832d4cfd8bc.slice/crio-bfe772be3a8e69ed303363c51e8ab064805415021e3b43bee1b9840526ac56fc WatchSource:0}: Error finding container bfe772be3a8e69ed303363c51e8ab064805415021e3b43bee1b9840526ac56fc: Status 404 returned error can't find the container with id bfe772be3a8e69ed303363c51e8ab064805415021e3b43bee1b9840526ac56fc Jul 24 22:27:59.955848 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:27:59.955813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6b445f89-kjqvv" event={"ID":"c3953506-d593-4f75-8458-6832d4cfd8bc","Type":"ContainerStarted","Data":"bfe772be3a8e69ed303363c51e8ab064805415021e3b43bee1b9840526ac56fc"} Jul 24 22:28:00.550740 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:00.550677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:28:00.642217 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:00.642178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:28:00.966823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:00.966789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6b445f89-kjqvv" event={"ID":"c3953506-d593-4f75-8458-6832d4cfd8bc","Type":"ContainerStarted","Data":"d843946526e5ff13fc4d10c6f02351b937df955b5143c006970af3577c605fc8"} Jul 24 22:28:00.983383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:00.983324 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f6b445f89-kjqvv" podStartSLOduration=1.608734782 podStartE2EDuration="1.983286907s" totalImagesPullingTime="374.552125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:28:00.9823225 +0000 UTC m=+788.344761821" watchObservedRunningTime="2026-07-24 22:28:00.983286907 +0000 UTC m=+788.345726227" Jul 24 22:28:01.019317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.019270 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:28:01.019595 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.019560 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" podUID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" containerName="authorino" containerID="cri-o://c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5" gracePeriod=30 Jul 24 22:28:01.284897 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.284868 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:28:01.444630 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.444590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwc4n\" (UniqueName: \"kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n\") pod \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " Jul 24 22:28:01.444857 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.444702 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert\") pod \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\" (UID: \"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66\") " Jul 24 22:28:01.448472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.448436 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n" (OuterVolumeSpecName: "kube-api-access-rwc4n") pod "32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" (UID: "32a1e8ff-7ad4-4ea5-883c-1a1af5733a66"). InnerVolumeSpecName "kube-api-access-rwc4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:28:01.458391 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.458341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" (UID: "32a1e8ff-7ad4-4ea5-883c-1a1af5733a66"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:28:01.545990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.545903 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc4n\" (UniqueName: \"kubernetes.io/projected/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-kube-api-access-rwc4n\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:28:01.545990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.545937 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66-tls-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:28:01.974600 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.974562 2576 generic.go:356] "Generic (PLEG): container finished" podID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" containerID="c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5" exitCode=0 Jul 24 22:28:01.975057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.974610 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" Jul 24 22:28:01.975057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.974652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" event={"ID":"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66","Type":"ContainerDied","Data":"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5"} Jul 24 22:28:01.975057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.974695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bc4bbc76-zdnkk" event={"ID":"32a1e8ff-7ad4-4ea5-883c-1a1af5733a66","Type":"ContainerDied","Data":"01c073ca00bb0e5a47a592140bc5db8e01badb237506d3f95392854afc63d346"} Jul 24 22:28:01.975057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.974718 2576 scope.go:117] "RemoveContainer" containerID="c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5" Jul 24 22:28:01.984264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.984240 2576 scope.go:117] "RemoveContainer" containerID="c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5" Jul 24 22:28:01.984550 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:28:01.984531 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5\": container with ID starting with c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5 not found: ID does not exist" containerID="c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5" Jul 24 22:28:01.984617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:01.984560 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5"} err="failed to get container status \"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5\": rpc error: code = NotFound desc = could not find container \"c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5\": container with ID starting with c6a56e4ceeb7bed7ebb0a969aa9092b2a7d22d2da58da64f4ecfc94a15f9f5b5 not found: ID does not exist" Jul 24 22:28:02.000917 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:02.000861 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:28:02.003371 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:02.003341 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85bc4bbc76-zdnkk"] Jul 24 22:28:03.182135 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:03.182102 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" path="/var/lib/kubelet/pods/32a1e8ff-7ad4-4ea5-883c-1a1af5733a66/volumes" Jul 24 22:28:46.117315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.117265 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:28:46.117861 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.117500 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" podUID="aa294a92-0036-48fe-a77a-a5b455a482f5" containerName="manager" containerID="cri-o://1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8" gracePeriod=30 Jul 24 22:28:46.376130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.376071 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:28:46.484709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.484673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xvtp\" (UniqueName: \"kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp\") pod \"aa294a92-0036-48fe-a77a-a5b455a482f5\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " Jul 24 22:28:46.484709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.484723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert\") pod \"aa294a92-0036-48fe-a77a-a5b455a482f5\" (UID: \"aa294a92-0036-48fe-a77a-a5b455a482f5\") " Jul 24 22:28:46.486889 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.486859 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp" (OuterVolumeSpecName: "kube-api-access-5xvtp") pod "aa294a92-0036-48fe-a77a-a5b455a482f5" (UID: "aa294a92-0036-48fe-a77a-a5b455a482f5"). InnerVolumeSpecName "kube-api-access-5xvtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:28:46.487042 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.487021 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aa294a92-0036-48fe-a77a-a5b455a482f5" (UID: "aa294a92-0036-48fe-a77a-a5b455a482f5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:28:46.585501 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.585464 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvtp\" (UniqueName: \"kubernetes.io/projected/aa294a92-0036-48fe-a77a-a5b455a482f5-kube-api-access-5xvtp\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:28:46.585501 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:46.585496 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa294a92-0036-48fe-a77a-a5b455a482f5-webhook-cert\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:28:47.219280 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.219247 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa294a92-0036-48fe-a77a-a5b455a482f5" containerID="1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8" exitCode=0 Jul 24 22:28:47.219280 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.219282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" event={"ID":"aa294a92-0036-48fe-a77a-a5b455a482f5","Type":"ContainerDied","Data":"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8"} Jul 24 22:28:47.219768 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.219327 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" Jul 24 22:28:47.219768 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.219345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656df4d8c4-fcwwr" event={"ID":"aa294a92-0036-48fe-a77a-a5b455a482f5","Type":"ContainerDied","Data":"14bf338d509b55d758f7797c15abcbe4ec030cb37a64877edb0b5736aaa033ab"} Jul 24 22:28:47.219768 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.219367 2576 scope.go:117] "RemoveContainer" containerID="1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8" Jul 24 22:28:47.228612 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.228595 2576 scope.go:117] "RemoveContainer" containerID="1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8" Jul 24 22:28:47.228889 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:28:47.228864 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8\": container with ID starting with 1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8 not found: ID does not exist" containerID="1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8" Jul 24 22:28:47.228991 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.228891 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8"} err="failed to get container status \"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8\": rpc error: code = NotFound desc = could not find container \"1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8\": container with ID starting with 1b06f880726b4a86d028ac31c408519040f3ba8190fc6ac2d3240031aa1911e8 not found: ID does not exist" Jul 24 22:28:47.238796 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.238771 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:28:47.242393 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:47.242369 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-656df4d8c4-fcwwr"] Jul 24 22:28:49.181654 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:28:49.181617 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa294a92-0036-48fe-a77a-a5b455a482f5" path="/var/lib/kubelet/pods/aa294a92-0036-48fe-a77a-a5b455a482f5/volumes" Jul 24 22:29:53.087129 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:29:53.087053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:29:53.087690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:29:53.087144 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:29:53.094119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:29:53.094099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:29:53.094354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:29:53.094337 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:30:06.508258 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:06.508224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:07.694510 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:07.694475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:43.921909 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:43.921881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:48.123174 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:48.123140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:48.221946 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:48.221906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:48.333752 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:48.333712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:52.746116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:52.746080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:54.201045 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:54.201010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:54.268149 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:54.268108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:54.349211 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:54.349180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:54.444990 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:54.444960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:30:54.544317 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:30:54.544218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:31:00.240923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:31:00.240889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:31:00.304027 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:31:00.303986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:31:32.695559 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:31:32.695523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:31:32.753236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:31:32.753203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:33:14.351583 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:33:14.351544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:33:14.442389 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:33:14.442355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:34:53.120139 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:53.120057 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:34:53.122043 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:53.122022 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:34:53.129952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:53.129931 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:34:53.131853 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:53.131837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:34:54.154051 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:54.154019 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:34:54.154460 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:54.154237 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" podUID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" containerName="manager" containerID="cri-o://72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113" gracePeriod=10 Jul 24 22:34:57.919952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:57.919926 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:34:58.010359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.010237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume\") pod \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " Jul 24 22:34:58.010359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.010290 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zl4w\" (UniqueName: \"kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w\") pod \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\" (UID: \"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e\") " Jul 24 22:34:58.010638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.010612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" (UID: "8cf8d657-5b0c-4a6f-93cf-4fad5b66300e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:34:58.012475 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.012453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w" (OuterVolumeSpecName: "kube-api-access-5zl4w") pod "8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" (UID: "8cf8d657-5b0c-4a6f-93cf-4fad5b66300e"). InnerVolumeSpecName "kube-api-access-5zl4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:34:58.111668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.111632 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-extensions-socket-volume\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:34:58.111668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.111664 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zl4w\" (UniqueName: \"kubernetes.io/projected/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e-kube-api-access-5zl4w\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:34:58.214771 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.214741 2576 generic.go:356] "Generic (PLEG): container finished" podID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" containerID="72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113" exitCode=0 Jul 24 22:34:58.214946 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.214780 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" Jul 24 22:34:58.214946 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.214865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" event={"ID":"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e","Type":"ContainerDied","Data":"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113"} Jul 24 22:34:58.214946 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.214910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv" event={"ID":"8cf8d657-5b0c-4a6f-93cf-4fad5b66300e","Type":"ContainerDied","Data":"0a2b3956368abc591880274c9ad652c57c945c6fb7f189045f7af366016aa2f8"} Jul 24 22:34:58.214946 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.214937 2576 scope.go:117] "RemoveContainer" containerID="72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113" Jul 24 22:34:58.224903 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.224885 2576 scope.go:117] "RemoveContainer" containerID="72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113" Jul 24 22:34:58.225182 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:34:58.225164 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113\": container with ID starting with 72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113 not found: ID does not exist" containerID="72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113" Jul 24 22:34:58.225236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.225192 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113"} err="failed to get container status \"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113\": rpc error: code = NotFound desc = could not find container \"72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113\": container with ID starting with 72a7473d6abcd89a595623c1c1aeb8346630d15e024e856f93b09f7902084113 not found: ID does not exist" Jul 24 22:34:58.236226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.236196 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:34:58.239828 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:58.239801 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c64gv"] Jul 24 22:34:59.181502 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:34:59.181469 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" path="/var/lib/kubelet/pods/8cf8d657-5b0c-4a6f-93cf-4fad5b66300e/volumes" Jul 24 22:35:07.241772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:07.241735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:35:55.239637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.239606 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd"] Jul 24 22:35:55.240059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240029 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" containerName="authorino" Jul 24 22:35:55.240059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240040 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" containerName="authorino" Jul 24 22:35:55.240059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240051 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" containerName="manager" Jul 24 22:35:55.240059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240057 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" containerName="manager" Jul 24 22:35:55.240191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240067 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa294a92-0036-48fe-a77a-a5b455a482f5" containerName="manager" Jul 24 22:35:55.240191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240073 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa294a92-0036-48fe-a77a-a5b455a482f5" containerName="manager" Jul 24 22:35:55.240191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240141 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf8d657-5b0c-4a6f-93cf-4fad5b66300e" containerName="manager" Jul 24 22:35:55.240191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240148 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a1e8ff-7ad4-4ea5-883c-1a1af5733a66" containerName="authorino" Jul 24 22:35:55.240191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.240156 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa294a92-0036-48fe-a77a-a5b455a482f5" containerName="manager" Jul 24 22:35:55.244414 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.244394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.247105 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.247084 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b4g4c\"" Jul 24 22:35:55.255471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.255445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd"] Jul 24 22:35:55.332693 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.332653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbea030d-784e-422a-b694-200f53809491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.332886 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.332782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp8w\" (UniqueName: \"kubernetes.io/projected/cbea030d-784e-422a-b694-200f53809491-kube-api-access-7cp8w\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.433265 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.433230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp8w\" (UniqueName: \"kubernetes.io/projected/cbea030d-784e-422a-b694-200f53809491-kube-api-access-7cp8w\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.433472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.433278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbea030d-784e-422a-b694-200f53809491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.433710 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.433691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbea030d-784e-422a-b694-200f53809491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.441846 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.441822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp8w\" (UniqueName: \"kubernetes.io/projected/cbea030d-784e-422a-b694-200f53809491-kube-api-access-7cp8w\") pod \"kuadrant-operator-controller-manager-86878c769f-rmkfd\" (UID: \"cbea030d-784e-422a-b694-200f53809491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.555985 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.555891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:55.699918 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:55.699894 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd"] Jul 24 22:35:55.702967 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:35:55.702940 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea030d_784e_422a_b694_200f53809491.slice/crio-4d00230687a32d38d60e22239d7f4a93afe2d24ca53a387e07c965abefd7a961 WatchSource:0}: Error finding container 4d00230687a32d38d60e22239d7f4a93afe2d24ca53a387e07c965abefd7a961: Status 404 returned error can't find the container with id 4d00230687a32d38d60e22239d7f4a93afe2d24ca53a387e07c965abefd7a961 Jul 24 22:35:56.511763 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:56.511717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" event={"ID":"cbea030d-784e-422a-b694-200f53809491","Type":"ContainerStarted","Data":"fde7fed517e03eacaac26e4608080095900e132833c5b37bbc64ec72e015450c"} Jul 24 22:35:56.512150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:56.511776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" event={"ID":"cbea030d-784e-422a-b694-200f53809491","Type":"ContainerStarted","Data":"4d00230687a32d38d60e22239d7f4a93afe2d24ca53a387e07c965abefd7a961"} Jul 24 22:35:56.512150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:56.511808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:35:56.535609 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:35:56.535563 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" podStartSLOduration=1.5355428469999999 podStartE2EDuration="1.535542847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:35:56.532813083 +0000 UTC m=+1263.895252403" watchObservedRunningTime="2026-07-24 22:35:56.535542847 +0000 UTC m=+1263.897982168" Jul 24 22:36:07.519429 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:36:07.519391 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmkfd" Jul 24 22:37:24.798617 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:37:24.798522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:38:34.812217 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:38:34.812165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:38:45.631311 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:38:45.631266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:38:45.729593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:38:45.729533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:39:24.367086 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:24.367052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:39:25.801091 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:25.801051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:39:53.157675 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:53.157650 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:39:53.160151 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:53.160128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:39:53.164669 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:53.164647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:39:53.166983 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:39:53.166964 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:40:47.013135 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:40:47.013024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:40:47.114535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:40:47.114497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:42:40.837572 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:42:40.837534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:08.388130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:08.388093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:11.315390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:11.315356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:11.416545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:11.416504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:11.737722 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:11.737680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:17.368659 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:17.368572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:17.466892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:17.466859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:17.584213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:17.584179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:17.899785 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:17.899745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:18.008640 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:18.008611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.066219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.066114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.158079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.158045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.220533 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.220499 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:20.224743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.224710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.227464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.227443 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-64bba790-serving-cert\"" Jul 24 22:43:20.227668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.227645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:43:20.232188 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.232127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:20.263471 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.263435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.358502 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.358416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.358777 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.358757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.358826 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.358808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfw4\" (UniqueName: \"kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.358877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.358863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.460168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.460130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfw4\" (UniqueName: \"kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.460168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.460175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.460459 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.460254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.460923 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.460897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.462766 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.462748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.468784 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.468763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfw4\" (UniqueName: \"kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4\") pod \"maas-api-e2e-ait-64bba790-78d86f459c-7dfkx\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.538327 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.538276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:20.688439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.688415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:20.690404 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:43:20.690377 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c88e2e9_2e0c_4c09_af0f_682beb009b3b.slice/crio-8e3d5ce70edb2ff091a494bcc81a07e6a928ea3528063e9de72e4b73a5f98d2f WatchSource:0}: Error finding container 8e3d5ce70edb2ff091a494bcc81a07e6a928ea3528063e9de72e4b73a5f98d2f: Status 404 returned error can't find the container with id 8e3d5ce70edb2ff091a494bcc81a07e6a928ea3528063e9de72e4b73a5f98d2f Jul 24 22:43:20.691801 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.691784 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:43:20.709717 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.709686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.794164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.794127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:20.889492 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:20.889406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" event={"ID":"5c88e2e9-2e0c-4c09-af0f-682beb009b3b","Type":"ContainerStarted","Data":"8e3d5ce70edb2ff091a494bcc81a07e6a928ea3528063e9de72e4b73a5f98d2f"} Jul 24 22:43:21.896753 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:21.896700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" event={"ID":"5c88e2e9-2e0c-4c09-af0f-682beb009b3b","Type":"ContainerStarted","Data":"651caef39c5ef268e51825bbe56d6d9ff0a0b8a3aba4b3a15a2021a316c0d097"} Jul 24 22:43:21.897235 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:21.896929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:21.915061 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:21.915016 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" podStartSLOduration=1.255281426 podStartE2EDuration="1.915000885s" totalImagesPullingTime="659.719459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:43:21.914714786 +0000 UTC m=+1709.277154109" watchObservedRunningTime="2026-07-24 22:43:21.915000885 +0000 UTC m=+1709.277440205" Jul 24 22:43:27.909491 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:27.909461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:35.542583 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:35.542545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:35.643487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:35.643447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:45.547718 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.547683 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h"] Jul 24 22:43:45.551282 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.551267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:45.553669 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.553651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:43:45.557585 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.557562 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h"] Jul 24 22:43:45.690668 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.690626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64bba790-sdt9h\" (UID: \"a835eda5-7c0a-4769-8096-61fd662b925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:45.791887 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.791850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64bba790-sdt9h\" (UID: \"a835eda5-7c0a-4769-8096-61fd662b925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:45.792537 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.792515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64bba790-sdt9h\" (UID: \"a835eda5-7c0a-4769-8096-61fd662b925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:45.865637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:45.865550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:46.003865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:46.003806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h"] Jul 24 22:43:46.006506 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:43:46.006477 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda835eda5_7c0a_4769_8096_61fd662b925e.slice/crio-f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053 WatchSource:0}: Error finding container f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053: Status 404 returned error can't find the container with id f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053 Jul 24 22:43:46.027746 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:46.027714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" event={"ID":"a835eda5-7c0a-4769-8096-61fd662b925e","Type":"ContainerStarted","Data":"f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053"} Jul 24 22:43:49.049589 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:49.049495 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" event={"ID":"a835eda5-7c0a-4769-8096-61fd662b925e","Type":"ContainerStarted","Data":"e4b8e6d6ec2460c3251501e112ff0301fce6ebe8a6309c05d65618f9aa327af0"} Jul 24 22:43:49.065458 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:49.065407 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" podStartSLOduration=1.619685053 podStartE2EDuration="4.065390909s" totalImagesPullingTime="2.445705856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:43:49.063552303 +0000 UTC m=+1736.425991616" watchObservedRunningTime="2026-07-24 22:43:49.065390909 +0000 UTC m=+1736.427830229" Jul 24 22:43:50.056353 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:50.056286 2576 generic.go:356] "Generic (PLEG): container finished" podID="a835eda5-7c0a-4769-8096-61fd662b925e" containerID="e4b8e6d6ec2460c3251501e112ff0301fce6ebe8a6309c05d65618f9aa327af0" exitCode=0 Jul 24 22:43:50.056733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:50.056402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" event={"ID":"a835eda5-7c0a-4769-8096-61fd662b925e","Type":"ContainerDied","Data":"e4b8e6d6ec2460c3251501e112ff0301fce6ebe8a6309c05d65618f9aa327af0"} Jul 24 22:43:51.204379 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:51.204348 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:51.346007 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:51.345915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca\") pod \"a835eda5-7c0a-4769-8096-61fd662b925e\" (UID: \"a835eda5-7c0a-4769-8096-61fd662b925e\") " Jul 24 22:43:51.346243 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:51.346220 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a835eda5-7c0a-4769-8096-61fd662b925e" (UID: "a835eda5-7c0a-4769-8096-61fd662b925e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:43:51.446766 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:51.446728 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a835eda5-7c0a-4769-8096-61fd662b925e-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:43:52.068879 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:52.068851 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" Jul 24 22:43:52.069072 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:52.068884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h" event={"ID":"a835eda5-7c0a-4769-8096-61fd662b925e","Type":"ContainerDied","Data":"f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053"} Jul 24 22:43:52.069072 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:52.068927 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7e5d6c3bbd86a384f66542915f7549864ce18094c4c37a3f65e3e044bb36053" Jul 24 22:43:55.958708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:55.958678 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:55.959117 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:55.958964 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" podUID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" containerName="maas-api" containerID="cri-o://651caef39c5ef268e51825bbe56d6d9ff0a0b8a3aba4b3a15a2021a316c0d097" gracePeriod=30 Jul 24 22:43:55.983910 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:55.983881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:56.038966 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.038929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:56.090712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.090678 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" containerID="651caef39c5ef268e51825bbe56d6d9ff0a0b8a3aba4b3a15a2021a316c0d097" exitCode=0 Jul 24 22:43:56.090891 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.090753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" event={"ID":"5c88e2e9-2e0c-4c09-af0f-682beb009b3b","Type":"ContainerDied","Data":"651caef39c5ef268e51825bbe56d6d9ff0a0b8a3aba4b3a15a2021a316c0d097"} Jul 24 22:43:56.094681 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.094625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:43:56.212783 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.212716 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:56.293538 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.293511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nfw4\" (UniqueName: \"kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4\") pod \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " Jul 24 22:43:56.293686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.293616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls\") pod \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " Jul 24 22:43:56.293686 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.293635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca\") pod \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\" (UID: \"5c88e2e9-2e0c-4c09-af0f-682beb009b3b\") " Jul 24 22:43:56.294109 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.294076 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c88e2e9-2e0c-4c09-af0f-682beb009b3b" (UID: "5c88e2e9-2e0c-4c09-af0f-682beb009b3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:43:56.296001 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.295977 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c88e2e9-2e0c-4c09-af0f-682beb009b3b" (UID: "5c88e2e9-2e0c-4c09-af0f-682beb009b3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:43:56.296096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.296010 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4" (OuterVolumeSpecName: "kube-api-access-6nfw4") pod "5c88e2e9-2e0c-4c09-af0f-682beb009b3b" (UID: "5c88e2e9-2e0c-4c09-af0f-682beb009b3b"). InnerVolumeSpecName "kube-api-access-6nfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:43:56.394476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.394439 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:43:56.394476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.394474 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:43:56.394683 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:56.394488 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfw4\" (UniqueName: \"kubernetes.io/projected/5c88e2e9-2e0c-4c09-af0f-682beb009b3b-kube-api-access-6nfw4\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:43:57.097510 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.097478 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" Jul 24 22:43:57.098015 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.097513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx" event={"ID":"5c88e2e9-2e0c-4c09-af0f-682beb009b3b","Type":"ContainerDied","Data":"8e3d5ce70edb2ff091a494bcc81a07e6a928ea3528063e9de72e4b73a5f98d2f"} Jul 24 22:43:57.098015 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.097580 2576 scope.go:117] "RemoveContainer" containerID="651caef39c5ef268e51825bbe56d6d9ff0a0b8a3aba4b3a15a2021a316c0d097" Jul 24 22:43:57.119308 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.119269 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:57.125084 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.125060 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64bba790-78d86f459c-7dfkx"] Jul 24 22:43:57.185389 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:43:57.184737 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" path="/var/lib/kubelet/pods/5c88e2e9-2e0c-4c09-af0f-682beb009b3b/volumes" Jul 24 22:44:00.626998 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:00.626962 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h"] Jul 24 22:44:00.628847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:00.628824 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64bba790-sdt9h"] Jul 24 22:44:00.632893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:00.632868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:00.772659 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:00.772625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:01.000479 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.000447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:01.182113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.182077 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a835eda5-7c0a-4769-8096-61fd662b925e" path="/var/lib/kubelet/pods/a835eda5-7c0a-4769-8096-61fd662b925e/volumes" Jul 24 22:44:01.225847 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.225820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:01.330170 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.330073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:01.428425 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.428389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:01.755777 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:01.755741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:02.109277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:02.109196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:04.413824 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.413787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:04.483165 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.483131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:04.484355 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.484326 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a835eda5-7c0a-4769-8096-61fd662b925e" containerName="revoke-keys" Jul 24 22:44:04.484530 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.484518 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a835eda5-7c0a-4769-8096-61fd662b925e" containerName="revoke-keys" Jul 24 22:44:04.484676 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.484665 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" containerName="maas-api" Jul 24 22:44:04.484752 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.484743 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" containerName="maas-api" Jul 24 22:44:04.485059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.485043 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a835eda5-7c0a-4769-8096-61fd662b925e" containerName="revoke-keys" Jul 24 22:44:04.485176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.485165 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c88e2e9-2e0c-4c09-af0f-682beb009b3b" containerName="maas-api" Jul 24 22:44:04.494418 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.494390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.497535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.497508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:04.497916 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.497894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1bab8920-serving-cert\"" Jul 24 22:44:04.498065 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.498047 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:44:04.517201 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.517165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:04.634570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.634536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:04.677839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.677740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hfr\" (UniqueName: \"kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.677839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.677808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.678048 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.677928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.778962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.778925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.779150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.778983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hfr\" (UniqueName: \"kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.779150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.779040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.779713 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.779681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.781599 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.781575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.787487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.787465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hfr\" (UniqueName: \"kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr\") pod \"maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.809357 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.809326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:04.961963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:04.961928 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:04.965876 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:44:04.965839 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4a641f_f58d_4002_8126_4e8b5079bc01.slice/crio-f920f958f1dedb64b067b988515988f05b97e1e7340432c5e5cf0c35bddd0474 WatchSource:0}: Error finding container f920f958f1dedb64b067b988515988f05b97e1e7340432c5e5cf0c35bddd0474: Status 404 returned error can't find the container with id f920f958f1dedb64b067b988515988f05b97e1e7340432c5e5cf0c35bddd0474 Jul 24 22:44:05.003550 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:05.003520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:05.093120 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:05.093072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:05.142434 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:05.142397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" event={"ID":"2a4a641f-f58d-4002-8126-4e8b5079bc01","Type":"ContainerStarted","Data":"f920f958f1dedb64b067b988515988f05b97e1e7340432c5e5cf0c35bddd0474"} Jul 24 22:44:06.149369 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:06.149326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" event={"ID":"2a4a641f-f58d-4002-8126-4e8b5079bc01","Type":"ContainerStarted","Data":"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc"} Jul 24 22:44:06.149806 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:06.149448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:06.165274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:06.165223 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" podStartSLOduration=1.810048908 podStartE2EDuration="2.165203421s" totalImagesPullingTime="355.154513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:44:06.164214456 +0000 UTC m=+1753.526653770" watchObservedRunningTime="2026-07-24 22:44:06.165203421 +0000 UTC m=+1753.527642742" Jul 24 22:44:12.160420 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:12.160388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:21.212070 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:21.212033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:29.397434 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.397395 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s"] Jul 24 22:44:29.401105 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.401083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:29.404101 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.404077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:44:29.406511 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.406487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s"] Jul 24 22:44:29.515190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.515154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s\" (UID: \"ae3cea96-ac72-41d4-b5d8-c071ba3b108d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:29.616468 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.616428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s\" (UID: \"ae3cea96-ac72-41d4-b5d8-c071ba3b108d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:29.617074 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.617053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s\" (UID: \"ae3cea96-ac72-41d4-b5d8-c071ba3b108d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:29.715152 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.715114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:29.849006 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:29.848980 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s"] Jul 24 22:44:29.851582 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:44:29.851548 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3cea96_ac72_41d4_b5d8_c071ba3b108d.slice/crio-dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6 WatchSource:0}: Error finding container dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6: Status 404 returned error can't find the container with id dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6 Jul 24 22:44:30.282648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:30.282554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" event={"ID":"ae3cea96-ac72-41d4-b5d8-c071ba3b108d","Type":"ContainerStarted","Data":"3cea90518e2ecbfe71578a6049cd437a1c15a0cdd0221240b3fd136d7fc9eb9d"} Jul 24 22:44:30.282648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:30.282615 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" event={"ID":"ae3cea96-ac72-41d4-b5d8-c071ba3b108d","Type":"ContainerStarted","Data":"dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6"} Jul 24 22:44:30.297507 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:30.297460 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" podStartSLOduration=1.2974431370000001 podStartE2EDuration="1.297443137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:44:30.296261304 +0000 UTC m=+1777.658700636" watchObservedRunningTime="2026-07-24 22:44:30.297443137 +0000 UTC m=+1777.659882496" Jul 24 22:44:31.289706 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:31.289673 2576 generic.go:356] "Generic (PLEG): container finished" podID="ae3cea96-ac72-41d4-b5d8-c071ba3b108d" containerID="3cea90518e2ecbfe71578a6049cd437a1c15a0cdd0221240b3fd136d7fc9eb9d" exitCode=0 Jul 24 22:44:31.290112 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:31.289764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" event={"ID":"ae3cea96-ac72-41d4-b5d8-c071ba3b108d","Type":"ContainerDied","Data":"3cea90518e2ecbfe71578a6049cd437a1c15a0cdd0221240b3fd136d7fc9eb9d"} Jul 24 22:44:32.452486 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:32.452457 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:32.542949 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:32.542914 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca\") pod \"ae3cea96-ac72-41d4-b5d8-c071ba3b108d\" (UID: \"ae3cea96-ac72-41d4-b5d8-c071ba3b108d\") " Jul 24 22:44:32.543270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:32.543246 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae3cea96-ac72-41d4-b5d8-c071ba3b108d" (UID: "ae3cea96-ac72-41d4-b5d8-c071ba3b108d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:44:32.643835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:32.643746 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cea96-ac72-41d4-b5d8-c071ba3b108d-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:44:33.305096 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:33.305064 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" Jul 24 22:44:33.305269 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:33.305059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s" event={"ID":"ae3cea96-ac72-41d4-b5d8-c071ba3b108d","Type":"ContainerDied","Data":"dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6"} Jul 24 22:44:33.305269 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:33.305184 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc66bc889cc7231236acdd934a0252c86bc5e32678cf63aea9f263286fe3de6" Jul 24 22:44:39.712360 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.712321 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:39.712864 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.712607 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" podUID="2a4a641f-f58d-4002-8126-4e8b5079bc01" containerName="maas-api" containerID="cri-o://79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc" gracePeriod=30 Jul 24 22:44:39.733134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.733105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:39.796435 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.796399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:39.853064 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.853028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:39.977713 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:39.977656 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:40.115107 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.115064 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls\") pod \"2a4a641f-f58d-4002-8126-4e8b5079bc01\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " Jul 24 22:44:40.115336 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.115114 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca\") pod \"2a4a641f-f58d-4002-8126-4e8b5079bc01\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " Jul 24 22:44:40.115336 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.115248 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6hfr\" (UniqueName: \"kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr\") pod \"2a4a641f-f58d-4002-8126-4e8b5079bc01\" (UID: \"2a4a641f-f58d-4002-8126-4e8b5079bc01\") " Jul 24 22:44:40.115513 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.115490 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a4a641f-f58d-4002-8126-4e8b5079bc01" (UID: "2a4a641f-f58d-4002-8126-4e8b5079bc01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:44:40.115715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.115688 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4a641f-f58d-4002-8126-4e8b5079bc01-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:44:40.117404 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.117381 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr" (OuterVolumeSpecName: "kube-api-access-q6hfr") pod "2a4a641f-f58d-4002-8126-4e8b5079bc01" (UID: "2a4a641f-f58d-4002-8126-4e8b5079bc01"). InnerVolumeSpecName "kube-api-access-q6hfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:44:40.117510 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.117382 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a4a641f-f58d-4002-8126-4e8b5079bc01" (UID: "2a4a641f-f58d-4002-8126-4e8b5079bc01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:44:40.216793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.216756 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hfr\" (UniqueName: \"kubernetes.io/projected/2a4a641f-f58d-4002-8126-4e8b5079bc01-kube-api-access-q6hfr\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:44:40.216793 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.216787 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4a641f-f58d-4002-8126-4e8b5079bc01-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:44:40.344205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.344121 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a4a641f-f58d-4002-8126-4e8b5079bc01" containerID="79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc" exitCode=0 Jul 24 22:44:40.344205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.344167 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" Jul 24 22:44:40.344205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.344181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" event={"ID":"2a4a641f-f58d-4002-8126-4e8b5079bc01","Type":"ContainerDied","Data":"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc"} Jul 24 22:44:40.344492 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.344223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j" event={"ID":"2a4a641f-f58d-4002-8126-4e8b5079bc01","Type":"ContainerDied","Data":"f920f958f1dedb64b067b988515988f05b97e1e7340432c5e5cf0c35bddd0474"} Jul 24 22:44:40.344492 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.344245 2576 scope.go:117] "RemoveContainer" containerID="79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc" Jul 24 22:44:40.354266 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.354246 2576 scope.go:117] "RemoveContainer" containerID="79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc" Jul 24 22:44:40.354581 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:44:40.354548 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc\": container with ID starting with 79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc not found: ID does not exist" containerID="79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc" Jul 24 22:44:40.354681 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.354592 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc"} err="failed to get container status \"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc\": rpc error: code = NotFound desc = could not find container \"79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc\": container with ID starting with 79e52135fefed71c9f051e93b9d5280d1f93a92ef50ce93dabfb1fabe64deccc not found: ID does not exist" Jul 24 22:44:40.365331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.365307 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:40.368823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:40.368800 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1bab8920-7575d4d5b4-b7w9j"] Jul 24 22:44:41.181820 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:41.181787 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4a641f-f58d-4002-8126-4e8b5079bc01" path="/var/lib/kubelet/pods/2a4a641f-f58d-4002-8126-4e8b5079bc01/volumes" Jul 24 22:44:44.460053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:44.459968 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s"] Jul 24 22:44:44.463662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:44.463637 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1bab8920-pwn5s"] Jul 24 22:44:44.465425 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:44.465371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:44.621994 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:44.621957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:44.874728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:44.874638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:45.077189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:45.077159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:45.181738 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:45.181656 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3cea96-ac72-41d4-b5d8-c071ba3b108d" path="/var/lib/kubelet/pods/ae3cea96-ac72-41d4-b5d8-c071ba3b108d/volumes" Jul 24 22:44:45.387278 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:45.387242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:45.519587 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:45.519542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:47.548427 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.548393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:47.612443 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.612407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:47.617284 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.617252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:44:47.617953 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.617927 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3cea96-ac72-41d4-b5d8-c071ba3b108d" containerName="revoke-keys" Jul 24 22:44:47.617953 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.617954 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3cea96-ac72-41d4-b5d8-c071ba3b108d" containerName="revoke-keys" Jul 24 22:44:47.618130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.617995 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4a641f-f58d-4002-8126-4e8b5079bc01" containerName="maas-api" Jul 24 22:44:47.618130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.618004 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4a641f-f58d-4002-8126-4e8b5079bc01" containerName="maas-api" Jul 24 22:44:47.618130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.618113 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3cea96-ac72-41d4-b5d8-c071ba3b108d" containerName="revoke-keys" Jul 24 22:44:47.618130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.618127 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4a641f-f58d-4002-8126-4e8b5079bc01" containerName="maas-api" Jul 24 22:44:47.622674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.622657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.625343 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.625318 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2db675aa-serving-cert\"" Jul 24 22:44:47.625474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.625394 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:44:47.632453 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.632429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:44:47.680247 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.680214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:47.790069 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.790032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.790251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.790122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfflk\" (UniqueName: \"kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.790251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.790153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.885903 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.885793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:47.890886 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.890866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfflk\" (UniqueName: \"kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.890974 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.890902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.890974 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.890944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.891558 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.891539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.893566 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.893543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.899892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.899875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfflk\" (UniqueName: \"kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk\") pod \"maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:47.935218 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:47.935189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:48.080362 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:48.080338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:44:48.082041 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:44:48.082006 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d02ffb_ac36_427d_964f_ecef5c73b532.slice/crio-3c4b20ad46514c9ddb98e4c53e7ae5f1a2306f76de89fd188e3644b85477c1aa WatchSource:0}: Error finding container 3c4b20ad46514c9ddb98e4c53e7ae5f1a2306f76de89fd188e3644b85477c1aa: Status 404 returned error can't find the container with id 3c4b20ad46514c9ddb98e4c53e7ae5f1a2306f76de89fd188e3644b85477c1aa Jul 24 22:44:48.098237 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:48.098184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:48.207398 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:48.207355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:44:48.390120 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:48.390089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" event={"ID":"57d02ffb-ac36-427d-964f-ecef5c73b532","Type":"ContainerStarted","Data":"3c4b20ad46514c9ddb98e4c53e7ae5f1a2306f76de89fd188e3644b85477c1aa"} Jul 24 22:44:49.396715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:49.396680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" event={"ID":"57d02ffb-ac36-427d-964f-ecef5c73b532","Type":"ContainerStarted","Data":"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978"} Jul 24 22:44:49.397126 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:49.396823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:44:49.412654 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:49.412602 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" podStartSLOduration=2.031111192 podStartE2EDuration="2.4125714s" totalImagesPullingTime="381.460208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:44:49.410880684 +0000 UTC m=+1796.773320004" watchObservedRunningTime="2026-07-24 22:44:49.4125714 +0000 UTC m=+1796.775010720" Jul 24 22:44:53.190497 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:53.190463 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:44:53.194210 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:53.194186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:44:53.197774 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:53.197753 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:44:53.201285 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:53.201268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:44:55.408088 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:44:55.408053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:45:00.130675 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.130636 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x"] Jul 24 22:45:00.134066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.134047 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7"] Jul 24 22:45:00.134178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.134159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" Jul 24 22:45:00.136623 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.136602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:45:00.137475 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.137444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" Jul 24 22:45:00.140931 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.140892 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x"] Jul 24 22:45:00.143926 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.143807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7"] Jul 24 22:45:00.148767 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.148743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" Jul 24 22:45:00.155395 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.155375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" Jul 24 22:45:00.303568 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.303283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x"] Jul 24 22:45:00.306968 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:45:00.306821 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496cb305_14c2_43a3_9492_0dd2fde1c2e5.slice/crio-64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1 WatchSource:0}: Error finding container 64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1: Status 404 returned error can't find the container with id 64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1 Jul 24 22:45:00.331937 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.331877 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7"] Jul 24 22:45:00.333937 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:45:00.333897 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda267a388_9222_4343_b4c6_f25503732bbd.slice/crio-ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba WatchSource:0}: Error finding container ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba: Status 404 returned error can't find the container with id ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba Jul 24 22:45:00.455053 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.455019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" event={"ID":"496cb305-14c2-43a3-9492-0dd2fde1c2e5","Type":"ContainerStarted","Data":"64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1"} Jul 24 22:45:00.456235 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:00.456213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" event={"ID":"a267a388-9222-4343-b4c6-f25503732bbd","Type":"ContainerStarted","Data":"ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba"} Jul 24 22:45:01.463416 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:01.463382 2576 generic.go:356] "Generic (PLEG): container finished" podID="496cb305-14c2-43a3-9492-0dd2fde1c2e5" containerID="6334a705a14afd973bb189760b6a1d0c15f1efc0ef3b8ec8e07e5f8263a59473" exitCode=0 Jul 24 22:45:01.463822 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:01.463467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" event={"ID":"496cb305-14c2-43a3-9492-0dd2fde1c2e5","Type":"ContainerDied","Data":"6334a705a14afd973bb189760b6a1d0c15f1efc0ef3b8ec8e07e5f8263a59473"} Jul 24 22:45:01.465041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:01.465016 2576 generic.go:356] "Generic (PLEG): container finished" podID="a267a388-9222-4343-b4c6-f25503732bbd" containerID="6b734965f68d2568f2268e731d873320416a0bf2209c20e808b33b253553bbd3" exitCode=0 Jul 24 22:45:01.465151 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:01.465062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" event={"ID":"a267a388-9222-4343-b4c6-f25503732bbd","Type":"ContainerDied","Data":"6b734965f68d2568f2268e731d873320416a0bf2209c20e808b33b253553bbd3"} Jul 24 22:45:02.624075 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:02.624053 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" Jul 24 22:45:02.645225 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:02.645194 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" Jul 24 22:45:03.476843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.476810 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" Jul 24 22:45:03.477035 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.476823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748885-2946x" event={"ID":"496cb305-14c2-43a3-9492-0dd2fde1c2e5","Type":"ContainerDied","Data":"64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1"} Jul 24 22:45:03.477035 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.476944 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ffe30c8d8b77308210e96f7bbf461fd5611e48d5ce58c03d7d99b7b4ead2d1" Jul 24 22:45:03.478241 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.478217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" event={"ID":"a267a388-9222-4343-b4c6-f25503732bbd","Type":"ContainerDied","Data":"ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba"} Jul 24 22:45:03.478241 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.478233 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7" Jul 24 22:45:03.478401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:03.478257 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb78ace2b4f814674ea84ee0781a0660fa2b94cc13acde436d4efcc1681a1ba" Jul 24 22:45:13.272743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.272707 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9"] Jul 24 22:45:13.273131 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273120 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496cb305-14c2-43a3-9492-0dd2fde1c2e5" containerName="cleanup" Jul 24 22:45:13.273131 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273131 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="496cb305-14c2-43a3-9492-0dd2fde1c2e5" containerName="cleanup" Jul 24 22:45:13.273205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273160 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a267a388-9222-4343-b4c6-f25503732bbd" containerName="cleanup" Jul 24 22:45:13.273205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273166 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a267a388-9222-4343-b4c6-f25503732bbd" containerName="cleanup" Jul 24 22:45:13.273286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273228 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a267a388-9222-4343-b4c6-f25503732bbd" containerName="cleanup" Jul 24 22:45:13.273286 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.273240 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="496cb305-14c2-43a3-9492-0dd2fde1c2e5" containerName="cleanup" Jul 24 22:45:13.276424 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.276407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:13.278795 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.278771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:45:13.282797 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.282770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9"] Jul 24 22:45:13.325146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.325110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9\" (UID: \"130c3e49-c303-4335-b0c5-d02ffdbe2eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:13.425638 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.425609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9\" (UID: \"130c3e49-c303-4335-b0c5-d02ffdbe2eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:13.426285 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.426264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9\" (UID: \"130c3e49-c303-4335-b0c5-d02ffdbe2eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:13.588656 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.588559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:13.730579 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:13.730556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9"] Jul 24 22:45:13.732974 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:45:13.732950 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130c3e49_c303_4335_b0c5_d02ffdbe2eb0.slice/crio-cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0 WatchSource:0}: Error finding container cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0: Status 404 returned error can't find the container with id cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0 Jul 24 22:45:14.544479 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:14.544445 2576 generic.go:356] "Generic (PLEG): container finished" podID="130c3e49-c303-4335-b0c5-d02ffdbe2eb0" containerID="665eea095a8074f6061f683e7e165c16e3b1294ddf65c03fd6d09b96492c60ce" exitCode=0 Jul 24 22:45:14.544905 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:14.544535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" event={"ID":"130c3e49-c303-4335-b0c5-d02ffdbe2eb0","Type":"ContainerDied","Data":"665eea095a8074f6061f683e7e165c16e3b1294ddf65c03fd6d09b96492c60ce"} Jul 24 22:45:14.544905 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:14.544590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" event={"ID":"130c3e49-c303-4335-b0c5-d02ffdbe2eb0","Type":"ContainerStarted","Data":"cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0"} Jul 24 22:45:15.689173 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:15.689148 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:15.748170 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:15.748136 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca\") pod \"130c3e49-c303-4335-b0c5-d02ffdbe2eb0\" (UID: \"130c3e49-c303-4335-b0c5-d02ffdbe2eb0\") " Jul 24 22:45:15.748573 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:15.748552 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "130c3e49-c303-4335-b0c5-d02ffdbe2eb0" (UID: "130c3e49-c303-4335-b0c5-d02ffdbe2eb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:45:15.849199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:15.849114 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130c3e49-c303-4335-b0c5-d02ffdbe2eb0-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:45:16.556386 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:16.556354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" event={"ID":"130c3e49-c303-4335-b0c5-d02ffdbe2eb0","Type":"ContainerDied","Data":"cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0"} Jul 24 22:45:16.556386 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:16.556369 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9" Jul 24 22:45:16.556602 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:16.556407 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0394bee3054da32201da7ca62c2b1181613208dec283fa330e68569d5337e0" Jul 24 22:45:23.623138 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.623103 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:45:23.623644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.623328 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" podUID="57d02ffb-ac36-427d-964f-ecef5c73b532" containerName="maas-api" containerID="cri-o://c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978" gracePeriod=30 Jul 24 22:45:23.639175 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.639139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:23.641205 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.641178 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7"] Jul 24 22:45:23.645906 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.645880 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2db675aa-29748885-ngtc7"] Jul 24 22:45:23.705570 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.705537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:23.762156 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.762112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:23.907543 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:23.907515 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:45:24.038591 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.038544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfflk\" (UniqueName: \"kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk\") pod \"57d02ffb-ac36-427d-964f-ecef5c73b532\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " Jul 24 22:45:24.038776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.038623 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls\") pod \"57d02ffb-ac36-427d-964f-ecef5c73b532\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " Jul 24 22:45:24.038776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.038728 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca\") pod \"57d02ffb-ac36-427d-964f-ecef5c73b532\" (UID: \"57d02ffb-ac36-427d-964f-ecef5c73b532\") " Jul 24 22:45:24.039050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.039028 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57d02ffb-ac36-427d-964f-ecef5c73b532" (UID: "57d02ffb-ac36-427d-964f-ecef5c73b532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:45:24.040924 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.040901 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk" (OuterVolumeSpecName: "kube-api-access-cfflk") pod "57d02ffb-ac36-427d-964f-ecef5c73b532" (UID: "57d02ffb-ac36-427d-964f-ecef5c73b532"). InnerVolumeSpecName "kube-api-access-cfflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:45:24.040924 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.040910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57d02ffb-ac36-427d-964f-ecef5c73b532" (UID: "57d02ffb-ac36-427d-964f-ecef5c73b532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:45:24.140405 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.140330 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02ffb-ac36-427d-964f-ecef5c73b532-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:45:24.140405 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.140356 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02ffb-ac36-427d-964f-ecef5c73b532-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:45:24.140405 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.140366 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfflk\" (UniqueName: \"kubernetes.io/projected/57d02ffb-ac36-427d-964f-ecef5c73b532-kube-api-access-cfflk\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:45:24.605289 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.605254 2576 generic.go:356] "Generic (PLEG): container finished" podID="57d02ffb-ac36-427d-964f-ecef5c73b532" containerID="c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978" exitCode=0 Jul 24 22:45:24.605499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.605319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" Jul 24 22:45:24.605499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.605337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" event={"ID":"57d02ffb-ac36-427d-964f-ecef5c73b532","Type":"ContainerDied","Data":"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978"} Jul 24 22:45:24.605499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.605404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc" event={"ID":"57d02ffb-ac36-427d-964f-ecef5c73b532","Type":"ContainerDied","Data":"3c4b20ad46514c9ddb98e4c53e7ae5f1a2306f76de89fd188e3644b85477c1aa"} Jul 24 22:45:24.605499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.605427 2576 scope.go:117] "RemoveContainer" containerID="c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978" Jul 24 22:45:24.616240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.616222 2576 scope.go:117] "RemoveContainer" containerID="c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978" Jul 24 22:45:24.616520 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:45:24.616504 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978\": container with ID starting with c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978 not found: ID does not exist" containerID="c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978" Jul 24 22:45:24.616574 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.616529 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978"} err="failed to get container status \"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978\": rpc error: code = NotFound desc = could not find container \"c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978\": container with ID starting with c08081441eac470b859b786ae2cacf200d4f24fa2ede02612d8c28b9090d4978 not found: ID does not exist" Jul 24 22:45:24.627653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.627626 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:45:24.630664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:24.630642 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2db675aa-5cff75958b-xjpbc"] Jul 24 22:45:25.182776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:25.182737 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d02ffb-ac36-427d-964f-ecef5c73b532" path="/var/lib/kubelet/pods/57d02ffb-ac36-427d-964f-ecef5c73b532/volumes" Jul 24 22:45:25.183037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:25.183024 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a267a388-9222-4343-b4c6-f25503732bbd" path="/var/lib/kubelet/pods/a267a388-9222-4343-b4c6-f25503732bbd/volumes" Jul 24 22:45:28.336824 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:28.336772 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9"] Jul 24 22:45:28.339244 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:28.339217 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2db675aa-pcjq9"] Jul 24 22:45:28.346269 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:28.346244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:29.183118 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:29.183088 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130c3e49-c303-4335-b0c5-d02ffdbe2eb0" path="/var/lib/kubelet/pods/130c3e49-c303-4335-b0c5-d02ffdbe2eb0/volumes" Jul 24 22:45:29.491066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:29.491034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:29.711393 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:29.711359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:29.925569 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:29.925474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:30.283212 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:30.283176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:30.429418 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:30.429378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:32.879583 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.879540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:32.961452 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.961412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:32.963898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.963873 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:45:32.964548 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964529 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130c3e49-c303-4335-b0c5-d02ffdbe2eb0" containerName="revoke-keys" Jul 24 22:45:32.964590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964555 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="130c3e49-c303-4335-b0c5-d02ffdbe2eb0" containerName="revoke-keys" Jul 24 22:45:32.964590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964575 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d02ffb-ac36-427d-964f-ecef5c73b532" containerName="maas-api" Jul 24 22:45:32.964590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964584 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d02ffb-ac36-427d-964f-ecef5c73b532" containerName="maas-api" Jul 24 22:45:32.964751 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964737 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d02ffb-ac36-427d-964f-ecef5c73b532" containerName="maas-api" Jul 24 22:45:32.964785 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.964759 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="130c3e49-c303-4335-b0c5-d02ffdbe2eb0" containerName="revoke-keys" Jul 24 22:45:32.969445 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.969427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:32.972207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.972190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-869c695e-serving-cert\"" Jul 24 22:45:32.972476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.972463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:45:32.977625 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:32.977602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:45:33.030753 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.030713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:33.127041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.127003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.127206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.127121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.127206 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.127157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9k4t\" (UniqueName: \"kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.228015 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.227978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.228272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.228225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.228450 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:45:33.228386 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-serving-cert: secret "maas-api-e2e-derive-869c695e-serving-cert" not found Jul 24 22:45:33.228582 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:45:33.228565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls podName:2c509fc1-8aa5-420c-ba6b-08884f549caa nodeName:}" failed. No retries permitted until 2026-07-24 22:45:33.728539908 +0000 UTC m=+1841.090979214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls") pod "maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" (UID: "2c509fc1-8aa5-420c-ba6b-08884f549caa") : secret "maas-api-e2e-derive-869c695e-serving-cert" not found Jul 24 22:45:33.228642 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.228617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k4t\" (UniqueName: \"kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.228729 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.228711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.240338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.240313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k4t\" (UniqueName: \"kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.440029 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.439976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:33.538969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.538869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:33.610554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.610523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:45:33.732653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.732615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.735097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.735078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") pod \"maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:33.881584 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:33.881495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:34.022613 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:34.022586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:45:34.025262 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:45:34.025234 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c509fc1_8aa5_420c_ba6b_08884f549caa.slice/crio-57ac693854cbd5c096dc7c62bfc6a0a8de36374112f17bebbcfa4ba645fdcb15 WatchSource:0}: Error finding container 57ac693854cbd5c096dc7c62bfc6a0a8de36374112f17bebbcfa4ba645fdcb15: Status 404 returned error can't find the container with id 57ac693854cbd5c096dc7c62bfc6a0a8de36374112f17bebbcfa4ba645fdcb15 Jul 24 22:45:34.661486 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:34.661455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" event={"ID":"2c509fc1-8aa5-420c-ba6b-08884f549caa","Type":"ContainerStarted","Data":"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0"} Jul 24 22:45:34.661670 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:34.661506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" event={"ID":"2c509fc1-8aa5-420c-ba6b-08884f549caa","Type":"ContainerStarted","Data":"57ac693854cbd5c096dc7c62bfc6a0a8de36374112f17bebbcfa4ba645fdcb15"} Jul 24 22:45:34.661670 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:34.661529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:34.678274 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:34.678223 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" podStartSLOduration=2.265605261 podStartE2EDuration="2.678207534s" totalImagesPullingTime="412.602273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:45:34.677472099 +0000 UTC m=+1842.039911420" watchObservedRunningTime="2026-07-24 22:45:34.678207534 +0000 UTC m=+1842.040646855" Jul 24 22:45:40.672582 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:40.672547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:45:57.631600 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.631547 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm"] Jul 24 22:45:57.635341 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.635320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:45:57.638034 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.638010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:45:57.641101 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.641079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm"] Jul 24 22:45:57.783068 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.783024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-869c695e-6xkkm\" (UID: \"3a56f41c-5a62-490f-94e0-2df16bc29889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:45:57.883676 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.883581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-869c695e-6xkkm\" (UID: \"3a56f41c-5a62-490f-94e0-2df16bc29889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:45:57.884226 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.884208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-869c695e-6xkkm\" (UID: \"3a56f41c-5a62-490f-94e0-2df16bc29889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:45:57.947648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:57.947598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:45:58.085008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:58.084975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm"] Jul 24 22:45:58.086856 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:45:58.086830 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a56f41c_5a62_490f_94e0_2df16bc29889.slice/crio-45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19 WatchSource:0}: Error finding container 45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19: Status 404 returned error can't find the container with id 45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19 Jul 24 22:45:58.794705 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:58.794667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" event={"ID":"3a56f41c-5a62-490f-94e0-2df16bc29889","Type":"ContainerStarted","Data":"23684e169878918651de508ddfb575b7a5a2bb84a350ff49976ed5716e0d96bf"} Jul 24 22:45:58.795155 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:58.794717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" event={"ID":"3a56f41c-5a62-490f-94e0-2df16bc29889","Type":"ContainerStarted","Data":"45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19"} Jul 24 22:45:58.811103 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:58.811048 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" podStartSLOduration=1.811027451 podStartE2EDuration="1.811027451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:45:58.809231997 +0000 UTC m=+1866.171671319" watchObservedRunningTime="2026-07-24 22:45:58.811027451 +0000 UTC m=+1866.173466772" Jul 24 22:45:59.802655 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:59.802623 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a56f41c-5a62-490f-94e0-2df16bc29889" containerID="23684e169878918651de508ddfb575b7a5a2bb84a350ff49976ed5716e0d96bf" exitCode=0 Jul 24 22:45:59.803012 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:45:59.802711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" event={"ID":"3a56f41c-5a62-490f-94e0-2df16bc29889","Type":"ContainerDied","Data":"23684e169878918651de508ddfb575b7a5a2bb84a350ff49976ed5716e0d96bf"} Jul 24 22:46:00.944442 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:00.944414 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:46:01.112563 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.112454 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca\") pod \"3a56f41c-5a62-490f-94e0-2df16bc29889\" (UID: \"3a56f41c-5a62-490f-94e0-2df16bc29889\") " Jul 24 22:46:01.112839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.112817 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a56f41c-5a62-490f-94e0-2df16bc29889" (UID: "3a56f41c-5a62-490f-94e0-2df16bc29889"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:46:01.213590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.213556 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a56f41c-5a62-490f-94e0-2df16bc29889-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:46:01.815439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.815405 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" Jul 24 22:46:01.815637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.815434 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm" event={"ID":"3a56f41c-5a62-490f-94e0-2df16bc29889","Type":"ContainerDied","Data":"45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19"} Jul 24 22:46:01.815637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:01.815490 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45aae97e7e4da7206f25880c6d85a3971c5a251929cd3c8e5efb551befe0ad19" Jul 24 22:46:08.012133 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.012100 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:46:08.012637 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.012361 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" podUID="2c509fc1-8aa5-420c-ba6b-08884f549caa" containerName="maas-api" containerID="cri-o://e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0" gracePeriod=30 Jul 24 22:46:08.056251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.056216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:08.133580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.133542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:08.194500 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.194459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:08.282822 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.282798 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:46:08.381267 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.381233 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9k4t\" (UniqueName: \"kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t\") pod \"2c509fc1-8aa5-420c-ba6b-08884f549caa\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " Jul 24 22:46:08.381476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.381327 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") pod \"2c509fc1-8aa5-420c-ba6b-08884f549caa\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " Jul 24 22:46:08.381476 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.381432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca\") pod \"2c509fc1-8aa5-420c-ba6b-08884f549caa\" (UID: \"2c509fc1-8aa5-420c-ba6b-08884f549caa\") " Jul 24 22:46:08.381805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.381784 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c509fc1-8aa5-420c-ba6b-08884f549caa" (UID: "2c509fc1-8aa5-420c-ba6b-08884f549caa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:46:08.383534 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.383509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c509fc1-8aa5-420c-ba6b-08884f549caa" (UID: "2c509fc1-8aa5-420c-ba6b-08884f549caa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:46:08.383534 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.383519 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t" (OuterVolumeSpecName: "kube-api-access-n9k4t") pod "2c509fc1-8aa5-420c-ba6b-08884f549caa" (UID: "2c509fc1-8aa5-420c-ba6b-08884f549caa"). InnerVolumeSpecName "kube-api-access-n9k4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:46:08.482514 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.482473 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c509fc1-8aa5-420c-ba6b-08884f549caa-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:46:08.482514 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.482504 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9k4t\" (UniqueName: \"kubernetes.io/projected/2c509fc1-8aa5-420c-ba6b-08884f549caa-kube-api-access-n9k4t\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:46:08.482514 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.482513 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c509fc1-8aa5-420c-ba6b-08884f549caa-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:46:08.854743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.854707 2576 generic.go:356] "Generic (PLEG): container finished" podID="2c509fc1-8aa5-420c-ba6b-08884f549caa" containerID="e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0" exitCode=0 Jul 24 22:46:08.854743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.854746 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" Jul 24 22:46:08.854956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.854763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" event={"ID":"2c509fc1-8aa5-420c-ba6b-08884f549caa","Type":"ContainerDied","Data":"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0"} Jul 24 22:46:08.854956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.854813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk" event={"ID":"2c509fc1-8aa5-420c-ba6b-08884f549caa","Type":"ContainerDied","Data":"57ac693854cbd5c096dc7c62bfc6a0a8de36374112f17bebbcfa4ba645fdcb15"} Jul 24 22:46:08.854956 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.854837 2576 scope.go:117] "RemoveContainer" containerID="e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0" Jul 24 22:46:08.864546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.864525 2576 scope.go:117] "RemoveContainer" containerID="e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0" Jul 24 22:46:08.864820 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:46:08.864801 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0\": container with ID starting with e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0 not found: ID does not exist" containerID="e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0" Jul 24 22:46:08.864877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.864829 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0"} err="failed to get container status \"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0\": rpc error: code = NotFound desc = could not find container \"e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0\": container with ID starting with e2df594d5ed5d81b24ac82ab285558831fb53e99fdd3e66ddb771b6b71a41ef0 not found: ID does not exist" Jul 24 22:46:08.876289 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.876265 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:46:08.880321 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:08.880280 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-869c695e-74bc7cbf6b-v7xmk"] Jul 24 22:46:09.182851 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:09.182766 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c509fc1-8aa5-420c-ba6b-08884f549caa" path="/var/lib/kubelet/pods/2c509fc1-8aa5-420c-ba6b-08884f549caa/volumes" Jul 24 22:46:12.703463 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:12.703428 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm"] Jul 24 22:46:12.705493 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:12.705466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:12.706268 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:12.706250 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-869c695e-6xkkm"] Jul 24 22:46:12.861884 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:12.861849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:13.103246 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:13.103148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:13.183263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:13.183233 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a56f41c-5a62-490f-94e0-2df16bc29889" path="/var/lib/kubelet/pods/3a56f41c-5a62-490f-94e0-2df16bc29889/volumes" Jul 24 22:46:13.585278 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:13.585234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:13.704104 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:13.704069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:15.103437 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:15.103405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:20.916992 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:20.916954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:20.990557 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:20.990519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:21.066717 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:21.066682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:21.167130 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:21.167036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:21.585648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:21.585614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:47.552679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:47.552640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:53.952869 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:53.952831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:54.015611 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:54.015577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:54.104387 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:54.104353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:46:54.213126 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:46:54.213028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:09.973176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:09.973136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:15.574019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:15.573987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:15.631141 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:15.631103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:15.698751 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:15.698716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:15.802826 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:15.802774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:16.835652 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:16.835620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:16.936199 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:16.936167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:26.273177 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:26.273139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:27.014606 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:27.014563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:28.044726 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:28.044683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:33.414023 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:33.413978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:33.517871 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:33.517825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:33.616036 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:33.616000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:33.715952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:33.715886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:34.436965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:34.436929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:34.534187 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:34.534152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:34.638950 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:34.638870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:36.214032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:36.213998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:41.888104 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:41.888060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:43.161612 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:43.161572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:48.446596 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:48.446563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:48.513473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:48.513434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:48.667936 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:48.667886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:48.973038 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:48.972994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:49.307805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:49.307713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.045805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.045756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.123773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.123737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.129290 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129250 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:47:52.129773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a56f41c-5a62-490f-94e0-2df16bc29889" containerName="revoke-keys" Jul 24 22:47:52.129773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129772 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a56f41c-5a62-490f-94e0-2df16bc29889" containerName="revoke-keys" Jul 24 22:47:52.129955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129788 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c509fc1-8aa5-420c-ba6b-08884f549caa" containerName="maas-api" Jul 24 22:47:52.129955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129794 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c509fc1-8aa5-420c-ba6b-08884f549caa" containerName="maas-api" Jul 24 22:47:52.129955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129856 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a56f41c-5a62-490f-94e0-2df16bc29889" containerName="revoke-keys" Jul 24 22:47:52.129955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.129866 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c509fc1-8aa5-420c-ba6b-08884f549caa" containerName="maas-api" Jul 24 22:47:52.133073 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.133051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.136070 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.136049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:47:52.136187 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.136127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85e566ec-serving-cert\"" Jul 24 22:47:52.143022 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.142993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:47:52.196530 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.196483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.257009 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.256958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.257221 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.257048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.257221 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.257118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zfp\" (UniqueName: \"kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.358277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.358181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.358277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.358234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zfp\" (UniqueName: \"kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.358509 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.358344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.358509 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:47:52.358383 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-serving-cert: secret "maas-api-e2e-mt-85e566ec-serving-cert" not found Jul 24 22:47:52.358509 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:47:52.358468 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls podName:57abfe2f-dc73-4936-b5c5-c1029c039165 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:52.858450293 +0000 UTC m=+1980.220889609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls") pod "maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" (UID: "57abfe2f-dc73-4936-b5c5-c1029c039165") : secret "maas-api-e2e-mt-85e566ec-serving-cert" not found Jul 24 22:47:52.359026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.359007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.372786 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.372741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zfp\" (UniqueName: \"kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.705670 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.705622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.803979 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.803947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:52.863496 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.863457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.866593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.866559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") pod \"maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:52.873843 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:52.873809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:53.048798 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:53.048715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:47:53.056605 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:53.056570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:53.216235 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:53.216209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:47:53.218573 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:47:53.218539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57abfe2f_dc73_4936_b5c5_c1029c039165.slice/crio-513fc97df34f2bdd1a29e1912794c0a5c4bac5f7feb36c12d9d39d82645e3230 WatchSource:0}: Error finding container 513fc97df34f2bdd1a29e1912794c0a5c4bac5f7feb36c12d9d39d82645e3230: Status 404 returned error can't find the container with id 513fc97df34f2bdd1a29e1912794c0a5c4bac5f7feb36c12d9d39d82645e3230 Jul 24 22:47:53.436410 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:53.436286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" event={"ID":"57abfe2f-dc73-4936-b5c5-c1029c039165","Type":"ContainerStarted","Data":"513fc97df34f2bdd1a29e1912794c0a5c4bac5f7feb36c12d9d39d82645e3230"} Jul 24 22:47:54.443168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:54.443124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" event={"ID":"57abfe2f-dc73-4936-b5c5-c1029c039165","Type":"ContainerStarted","Data":"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61"} Jul 24 22:47:54.443590 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:54.443194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:47:54.478351 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:54.478268 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" podStartSLOduration=2.039173161 podStartE2EDuration="2.47825005s" totalImagesPullingTime="439.076889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:54.477235451 +0000 UTC m=+1981.839674775" watchObservedRunningTime="2026-07-24 22:47:54.47825005 +0000 UTC m=+1981.840689371" Jul 24 22:47:56.523225 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:56.523181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:57.134695 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:57.134663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:57.237728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:57.237691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:47:57.554942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:47:57.554910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:00.454331 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:00.454280 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:48:03.337647 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.337596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:03.437978 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.437941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:03.491189 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.491141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:03.685960 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.685876 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6"] Jul 24 22:48:03.689622 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.689603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:03.692332 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.692312 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:48:03.699325 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.699283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6"] Jul 24 22:48:03.763982 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.763937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6\" (UID: \"232fe543-5e42-47de-b1e3-64c4be290072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:03.865059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.865016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6\" (UID: \"232fe543-5e42-47de-b1e3-64c4be290072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:03.865714 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:03.865696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6\" (UID: \"232fe543-5e42-47de-b1e3-64c4be290072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:04.001608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:04.001552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:04.144350 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:04.144215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6"] Jul 24 22:48:04.147482 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:48:04.147449 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232fe543_5e42_47de_b1e3_64c4be290072.slice/crio-aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe WatchSource:0}: Error finding container aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe: Status 404 returned error can't find the container with id aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe Jul 24 22:48:04.500132 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:04.500098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" event={"ID":"232fe543-5e42-47de-b1e3-64c4be290072","Type":"ContainerStarted","Data":"aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe"} Jul 24 22:48:05.507938 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:05.507901 2576 generic.go:356] "Generic (PLEG): container finished" podID="232fe543-5e42-47de-b1e3-64c4be290072" containerID="83afe36852bb57a0a3dbc1da3b86cf7bd0ec4224b7b1226daa53cce0ffaf94bb" exitCode=0 Jul 24 22:48:05.508376 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:05.507974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" event={"ID":"232fe543-5e42-47de-b1e3-64c4be290072","Type":"ContainerDied","Data":"83afe36852bb57a0a3dbc1da3b86cf7bd0ec4224b7b1226daa53cce0ffaf94bb"} Jul 24 22:48:06.656211 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:06.656186 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:06.795099 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:06.795007 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca\") pod \"232fe543-5e42-47de-b1e3-64c4be290072\" (UID: \"232fe543-5e42-47de-b1e3-64c4be290072\") " Jul 24 22:48:06.795440 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:06.795414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "232fe543-5e42-47de-b1e3-64c4be290072" (UID: "232fe543-5e42-47de-b1e3-64c4be290072"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:06.896359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:06.896290 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fe543-5e42-47de-b1e3-64c4be290072-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:48:07.521379 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:07.521335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" event={"ID":"232fe543-5e42-47de-b1e3-64c4be290072","Type":"ContainerDied","Data":"aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe"} Jul 24 22:48:07.521527 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:07.521420 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaff2787e06670bc9a00d3119f4f8ab7df42968e6c32921e0d93d45173c6b7fe" Jul 24 22:48:07.521527 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:07.521360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6" Jul 24 22:48:13.784487 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:13.784452 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:48:13.784962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:13.784706 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" podUID="57abfe2f-dc73-4936-b5c5-c1029c039165" containerName="maas-api" containerID="cri-o://05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61" gracePeriod=30 Jul 24 22:48:13.804754 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:13.804718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:13.876270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:13.876232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:13.934878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:13.934842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:14.044192 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.044127 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:48:14.068353 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.068324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89zfp\" (UniqueName: \"kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp\") pod \"57abfe2f-dc73-4936-b5c5-c1029c039165\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " Jul 24 22:48:14.068535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.068380 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") pod \"57abfe2f-dc73-4936-b5c5-c1029c039165\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " Jul 24 22:48:14.068535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.068447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca\") pod \"57abfe2f-dc73-4936-b5c5-c1029c039165\" (UID: \"57abfe2f-dc73-4936-b5c5-c1029c039165\") " Jul 24 22:48:14.068856 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.068828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57abfe2f-dc73-4936-b5c5-c1029c039165" (UID: "57abfe2f-dc73-4936-b5c5-c1029c039165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:14.070727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.070703 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp" (OuterVolumeSpecName: "kube-api-access-89zfp") pod "57abfe2f-dc73-4936-b5c5-c1029c039165" (UID: "57abfe2f-dc73-4936-b5c5-c1029c039165"). InnerVolumeSpecName "kube-api-access-89zfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:14.071030 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.070998 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57abfe2f-dc73-4936-b5c5-c1029c039165" (UID: "57abfe2f-dc73-4936-b5c5-c1029c039165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:14.169082 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.169045 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57abfe2f-dc73-4936-b5c5-c1029c039165-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:48:14.169271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.169099 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57abfe2f-dc73-4936-b5c5-c1029c039165-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:48:14.169271 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.169117 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89zfp\" (UniqueName: \"kubernetes.io/projected/57abfe2f-dc73-4936-b5c5-c1029c039165-kube-api-access-89zfp\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:48:14.560156 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.560116 2576 generic.go:356] "Generic (PLEG): container finished" podID="57abfe2f-dc73-4936-b5c5-c1029c039165" containerID="05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61" exitCode=0 Jul 24 22:48:14.560369 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.560165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" Jul 24 22:48:14.560369 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.560203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" event={"ID":"57abfe2f-dc73-4936-b5c5-c1029c039165","Type":"ContainerDied","Data":"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61"} Jul 24 22:48:14.560369 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.560261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v" event={"ID":"57abfe2f-dc73-4936-b5c5-c1029c039165","Type":"ContainerDied","Data":"513fc97df34f2bdd1a29e1912794c0a5c4bac5f7feb36c12d9d39d82645e3230"} Jul 24 22:48:14.560369 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.560283 2576 scope.go:117] "RemoveContainer" containerID="05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61" Jul 24 22:48:14.570083 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.570065 2576 scope.go:117] "RemoveContainer" containerID="05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61" Jul 24 22:48:14.570404 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:48:14.570383 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61\": container with ID starting with 05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61 not found: ID does not exist" containerID="05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61" Jul 24 22:48:14.570464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.570413 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61"} err="failed to get container status \"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61\": rpc error: code = NotFound desc = could not find container \"05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61\": container with ID starting with 05b7e4a52529b0948e0c6f0a022af6fdb2b3cffa96f431ce17c795fba5fb4a61 not found: ID does not exist" Jul 24 22:48:14.581893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.581868 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:48:14.584965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:14.584944 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e566ec-7dc5dd4c5b-6ch2v"] Jul 24 22:48:15.182963 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:15.182930 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57abfe2f-dc73-4936-b5c5-c1029c039165" path="/var/lib/kubelet/pods/57abfe2f-dc73-4936-b5c5-c1029c039165/volumes" Jul 24 22:48:18.738653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:18.738614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:18.748826 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:18.748794 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6"] Jul 24 22:48:18.756122 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:18.756094 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85e566ec-jtdl6"] Jul 24 22:48:19.095236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:19.095143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:19.181751 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:19.181717 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232fe543-5e42-47de-b1e3-64c4be290072" path="/var/lib/kubelet/pods/232fe543-5e42-47de-b1e3-64c4be290072/volumes" Jul 24 22:48:24.773145 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:24.773108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:24.872898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:24.872839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:25.021204 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:25.021158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:25.238002 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:25.237968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:25.348727 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:25.348695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:25.447344 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:25.447268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:25.566592 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:25.566488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:31.095122 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:31.095084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:31.196593 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:31.196552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:31.295615 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:31.295576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:31.420093 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:31.419998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:37.113878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:37.113826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:37.214600 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:37.214558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:43.822748 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:43.822703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:44.139955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:44.139863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:44.660867 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:44.660791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:47.284230 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.284173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:47.383507 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.383476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:47.511269 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511237 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:48:47.511679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511665 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fe543-5e42-47de-b1e3-64c4be290072" containerName="revoke-keys" Jul 24 22:48:47.511679 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511679 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fe543-5e42-47de-b1e3-64c4be290072" containerName="revoke-keys" Jul 24 22:48:47.511802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511702 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57abfe2f-dc73-4936-b5c5-c1029c039165" containerName="maas-api" Jul 24 22:48:47.511802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511708 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57abfe2f-dc73-4936-b5c5-c1029c039165" containerName="maas-api" Jul 24 22:48:47.511802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511777 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fe543-5e42-47de-b1e3-64c4be290072" containerName="revoke-keys" Jul 24 22:48:47.511802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.511787 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57abfe2f-dc73-4936-b5c5-c1029c039165" containerName="maas-api" Jul 24 22:48:47.515073 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.515053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.517911 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.517886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:48:47.518056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.517897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-540d8b1d-serving-cert\"" Jul 24 22:48:47.523611 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.523586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:48:47.542190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.542108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:47.585097 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.585063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wwb\" (UniqueName: \"kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.585365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.585128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.585365 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.585195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.686060 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.686027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wwb\" (UniqueName: \"kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.686251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.686098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.686251 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.686127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.686861 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.686827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.688745 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.688726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.698347 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.698318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wwb\" (UniqueName: \"kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb\") pod \"maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.723404 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.723369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:47.827525 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.827401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:47.990989 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.990966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:48:47.993178 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:48:47.993148 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2510d2_4fe0_46c4_a12b_59af47c219d3.slice/crio-819888772ab21163de4c16d62eed33131ebaa2e9e70bf6a8f9c5182026158987 WatchSource:0}: Error finding container 819888772ab21163de4c16d62eed33131ebaa2e9e70bf6a8f9c5182026158987: Status 404 returned error can't find the container with id 819888772ab21163de4c16d62eed33131ebaa2e9e70bf6a8f9c5182026158987 Jul 24 22:48:47.994597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:47.994578 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:48:48.006776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.006739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:48.098324 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.098192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:48.755473 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.755431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" event={"ID":"8a2510d2-4fe0-46c4-a12b-59af47c219d3","Type":"ContainerStarted","Data":"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9"} Jul 24 22:48:48.755962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.755527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" event={"ID":"8a2510d2-4fe0-46c4-a12b-59af47c219d3","Type":"ContainerStarted","Data":"819888772ab21163de4c16d62eed33131ebaa2e9e70bf6a8f9c5182026158987"} Jul 24 22:48:48.755962 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.755571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:48.774374 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:48.774322 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" podStartSLOduration=1.379652582 podStartE2EDuration="1.774284348s" totalImagesPullingTime="394.631766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:48:48.772160172 +0000 UTC m=+2036.134599494" watchObservedRunningTime="2026-07-24 22:48:48.774284348 +0000 UTC m=+2036.136723669" Jul 24 22:48:51.756235 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:51.756197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:48:54.766667 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:54.766635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:48:59.834987 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:48:59.834952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:30.267666 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:30.267586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:30.368614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:30.368578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:36.109263 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.109222 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk"] Jul 24 22:49:36.112880 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.112862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:36.115242 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.115221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:49:36.119961 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.119927 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk"] Jul 24 22:49:36.259597 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.259544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk\" (UID: \"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:36.361032 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.360935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk\" (UID: \"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:36.361677 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.361655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk\" (UID: \"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:36.426865 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.426829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:36.567693 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:36.567666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk"] Jul 24 22:49:36.570470 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:49:36.570420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ea26bb_4783_4e1a_8c0f_ec55b714f7ee.slice/crio-f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f WatchSource:0}: Error finding container f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f: Status 404 returned error can't find the container with id f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f Jul 24 22:49:37.032191 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:37.032150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" event={"ID":"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee","Type":"ContainerStarted","Data":"eccb1f271738a37a2f4734dc59b9d44a969a7dc3e05dcdd8a2889d9026fc5b32"} Jul 24 22:49:37.032384 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:37.032245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" event={"ID":"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee","Type":"ContainerStarted","Data":"f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f"} Jul 24 22:49:37.048573 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:37.048518 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" podStartSLOduration=1.048502741 podStartE2EDuration="1.048502741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:49:37.046460775 +0000 UTC m=+2084.408900097" watchObservedRunningTime="2026-07-24 22:49:37.048502741 +0000 UTC m=+2084.410942061" Jul 24 22:49:38.039367 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:38.039335 2576 generic.go:356] "Generic (PLEG): container finished" podID="f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" containerID="eccb1f271738a37a2f4734dc59b9d44a969a7dc3e05dcdd8a2889d9026fc5b32" exitCode=0 Jul 24 22:49:38.039755 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:38.039426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" event={"ID":"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee","Type":"ContainerDied","Data":"eccb1f271738a37a2f4734dc59b9d44a969a7dc3e05dcdd8a2889d9026fc5b32"} Jul 24 22:49:39.195359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:39.195331 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:39.289289 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:39.289251 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca\") pod \"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee\" (UID: \"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee\") " Jul 24 22:49:39.296221 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:39.289983 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" (UID: "f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:39.390596 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:39.390554 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:49:40.052057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:40.052026 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" Jul 24 22:49:40.052057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:40.052038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk" event={"ID":"f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee","Type":"ContainerDied","Data":"f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f"} Jul 24 22:49:40.052270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:40.052116 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f160ce7c4a808c99abd4a2e961c12fe8afd5e4a6e46c069d2f098f5d4eb9270f" Jul 24 22:49:51.455967 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.455924 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:49:51.456474 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.456231 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" podUID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" containerName="maas-api" containerID="cri-o://9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9" gracePeriod=30 Jul 24 22:49:51.470839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.470800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:51.571720 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.571690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:51.673190 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.673152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:51.716282 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.716217 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:49:51.808309 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.808270 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9wwb\" (UniqueName: \"kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb\") pod \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " Jul 24 22:49:51.808490 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.808348 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls\") pod \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " Jul 24 22:49:51.808490 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.808448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca\") pod \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\" (UID: \"8a2510d2-4fe0-46c4-a12b-59af47c219d3\") " Jul 24 22:49:51.808878 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.808839 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a2510d2-4fe0-46c4-a12b-59af47c219d3" (UID: "8a2510d2-4fe0-46c4-a12b-59af47c219d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:51.810903 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.810878 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb" (OuterVolumeSpecName: "kube-api-access-f9wwb") pod "8a2510d2-4fe0-46c4-a12b-59af47c219d3" (UID: "8a2510d2-4fe0-46c4-a12b-59af47c219d3"). InnerVolumeSpecName "kube-api-access-f9wwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:49:51.811011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.810932 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a2510d2-4fe0-46c4-a12b-59af47c219d3" (UID: "8a2510d2-4fe0-46c4-a12b-59af47c219d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:51.909499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.909462 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9wwb\" (UniqueName: \"kubernetes.io/projected/8a2510d2-4fe0-46c4-a12b-59af47c219d3-kube-api-access-f9wwb\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:49:51.909499 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.909496 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2510d2-4fe0-46c4-a12b-59af47c219d3-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:49:51.909715 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:51.909511 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2510d2-4fe0-46c4-a12b-59af47c219d3-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:49:52.126676 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.126578 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" containerID="9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9" exitCode=0 Jul 24 22:49:52.126676 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.126633 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" Jul 24 22:49:52.126676 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.126657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" event={"ID":"8a2510d2-4fe0-46c4-a12b-59af47c219d3","Type":"ContainerDied","Data":"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9"} Jul 24 22:49:52.126947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.126708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn" event={"ID":"8a2510d2-4fe0-46c4-a12b-59af47c219d3","Type":"ContainerDied","Data":"819888772ab21163de4c16d62eed33131ebaa2e9e70bf6a8f9c5182026158987"} Jul 24 22:49:52.126947 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.126753 2576 scope.go:117] "RemoveContainer" containerID="9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9" Jul 24 22:49:52.137134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.137108 2576 scope.go:117] "RemoveContainer" containerID="9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9" Jul 24 22:49:52.137450 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:49:52.137434 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9\": container with ID starting with 9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9 not found: ID does not exist" containerID="9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9" Jul 24 22:49:52.137518 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.137457 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9"} err="failed to get container status \"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9\": rpc error: code = NotFound desc = could not find container \"9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9\": container with ID starting with 9f098f88665371fec4b719d4fb1a153eccdedc1fe4cbc13d931068d3f9a28bc9 not found: ID does not exist" Jul 24 22:49:52.147536 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.147505 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:49:52.151608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:52.151583 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-540d8b1d-5fdb9f96c-kntxn"] Jul 24 22:49:53.182173 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.182141 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" path="/var/lib/kubelet/pods/8a2510d2-4fe0-46c4-a12b-59af47c219d3/volumes" Jul 24 22:49:53.227401 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.227363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:49:53.230838 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.230816 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:49:53.234390 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.234361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:49:53.237919 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.237901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:49:53.371669 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:53.371639 2576 scope.go:117] "RemoveContainer" containerID="e4b8e6d6ec2460c3251501e112ff0301fce6ebe8a6309c05d65618f9aa327af0" Jul 24 22:49:56.172276 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:56.172234 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk"] Jul 24 22:49:56.173772 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:56.173735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:49:56.174338 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:56.174319 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-540d8b1d-nj5mk"] Jul 24 22:49:57.183213 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:57.183173 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" path="/var/lib/kubelet/pods/f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee/volumes" Jul 24 22:49:59.245046 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:49:59.245012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:50:04.329019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:04.328990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:50:40.912778 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:40.912718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:50:41.279608 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:41.279562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:50:46.884240 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:46.884203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:50:53.397588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:53.397474 2576 scope.go:117] "RemoveContainer" containerID="3cea90518e2ecbfe71578a6049cd437a1c15a0cdd0221240b3fd136d7fc9eb9d" Jul 24 22:50:55.173066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:50:55.173018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:00.846055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:00.846022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:00.901146 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:00.901114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:01.200621 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:01.200586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:01.347239 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:01.347178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:02.389650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:02.389571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:02.487595 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:02.487549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:02.585058 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:02.585023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:03.291050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:03.291013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:03.390215 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:03.390179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:03.490272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:03.490217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:04.793255 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:04.793217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:10.804353 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:10.804314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:10.904712 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:10.904681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:11.459756 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:11.459717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:17.390792 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:17.390736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:17.451034 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:17.450997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:17.541040 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:17.541010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:17.677738 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:17.677637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:24.358361 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:24.358330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:24.407965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:24.407912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:28.092582 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:28.092550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:28.406142 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:28.406057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:49.338081 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:49.338046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:53.420656 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:53.420622 2576 scope.go:117] "RemoveContainer" containerID="665eea095a8074f6061f683e7e165c16e3b1294ddf65c03fd6d09b96492c60ce" Jul 24 22:51:53.429695 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:53.429675 2576 scope.go:117] "RemoveContainer" containerID="6b734965f68d2568f2268e731d873320416a0bf2209c20e808b33b253553bbd3" Jul 24 22:51:56.014150 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:56.014109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:56.114354 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:56.114289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:56.366106 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:56.365996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:56.664159 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:56.664059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:56.853388 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:56.853327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:59.239874 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.239840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:59.339000 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.338966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:59.418207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.418170 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:51:59.418893 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.418871 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" containerName="maas-api" Jul 24 22:51:59.418970 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.418897 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" containerName="maas-api" Jul 24 22:51:59.418970 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.418927 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" containerName="revoke-keys" Jul 24 22:51:59.418970 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.418937 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" containerName="revoke-keys" Jul 24 22:51:59.419078 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.419064 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ea26bb-4783-4e1a-8c0f-ec55b714f7ee" containerName="revoke-keys" Jul 24 22:51:59.419114 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.419084 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2510d2-4fe0-46c4-a12b-59af47c219d3" containerName="maas-api" Jul 24 22:51:59.422567 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.422547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.425026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.424993 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-11557e-serving-cert\"" Jul 24 22:51:59.425026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.425001 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:51:59.432641 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.432600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:51:59.450587 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.450551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:59.519449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.519348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmhm\" (UniqueName: \"kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.519449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.519426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.519722 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.519487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.583853 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.583813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:51:59.620265 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.620228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmhm\" (UniqueName: \"kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.620462 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.620338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.620462 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.620388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.621109 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.621086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.622903 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.622878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.630588 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.630566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmhm\" (UniqueName: \"kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm\") pod \"maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.734599 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.734562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:51:59.887746 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.887710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:51:59.892116 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:51:59.892076 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae33444_5b8c_4053_8fda_0662839cc808.slice/crio-f07d9bc823257413e0a248381055af073b01d5fed7d36f3fe5061412dced1941 WatchSource:0}: Error finding container f07d9bc823257413e0a248381055af073b01d5fed7d36f3fe5061412dced1941: Status 404 returned error can't find the container with id f07d9bc823257413e0a248381055af073b01d5fed7d36f3fe5061412dced1941 Jul 24 22:51:59.923350 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:51:59.923317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:00.012200 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:00.012158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:00.858132 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:00.858083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" event={"ID":"7ae33444-5b8c-4053-8fda-0662839cc808","Type":"ContainerStarted","Data":"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922"} Jul 24 22:52:00.858644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:00.858140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" event={"ID":"7ae33444-5b8c-4053-8fda-0662839cc808","Type":"ContainerStarted","Data":"f07d9bc823257413e0a248381055af073b01d5fed7d36f3fe5061412dced1941"} Jul 24 22:52:00.858644 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:00.858278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:52:00.875703 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:00.875653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" podStartSLOduration=1.378953807 podStartE2EDuration="1.875637864s" totalImagesPullingTime="496.684057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:00.873696608 +0000 UTC m=+2228.236135930" watchObservedRunningTime="2026-07-24 22:52:00.875637864 +0000 UTC m=+2228.238077185" Jul 24 22:52:03.736052 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:03.736016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:04.050753 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:04.050647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:04.340105 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:04.339981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:04.438027 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:04.437990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:06.868886 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:06.868857 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:52:07.375854 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.375820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:07.475232 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.475192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:07.638557 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.638483 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:52:07.642485 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.642464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.645329 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.645290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-422c4e-serving-cert\"" Jul 24 22:52:07.652995 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.652963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:52:07.686626 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.686595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:07.693955 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.693927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.694134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.693963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.694134 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.694039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsf6\" (UniqueName: \"kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.795623 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.795584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.795623 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.795626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.795863 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.795679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsf6\" (UniqueName: \"kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.795863 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:52:07.795746 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-serving-cert: secret "maas-api-e2e-inf-b-422c4e-serving-cert" not found Jul 24 22:52:07.795863 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:52:07.795819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls podName:d335e5cf-3ccf-43e6-887d-b40794ca93b6 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:08.295797655 +0000 UTC m=+2235.658236979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls") pod "maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" (UID: "d335e5cf-3ccf-43e6-887d-b40794ca93b6") : secret "maas-api-e2e-inf-b-422c4e-serving-cert" not found Jul 24 22:52:07.796312 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.796276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.805068 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.805040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsf6\" (UniqueName: \"kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:07.998030 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:07.997990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:08.164969 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.164928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:08.270688 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.270582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:08.301863 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.301823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:08.304519 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.304488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:08.555983 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.555889 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:08.696896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.696861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:52:08.698659 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:52:08.698627 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd335e5cf_3ccf_43e6_887d_b40794ca93b6.slice/crio-f0e85fd662fb176049cabc688cbc9d6192ebef711af93cb71d1cbec771367d03 WatchSource:0}: Error finding container f0e85fd662fb176049cabc688cbc9d6192ebef711af93cb71d1cbec771367d03: Status 404 returned error can't find the container with id f0e85fd662fb176049cabc688cbc9d6192ebef711af93cb71d1cbec771367d03 Jul 24 22:52:08.902156 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:08.902045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" event={"ID":"d335e5cf-3ccf-43e6-887d-b40794ca93b6","Type":"ContainerStarted","Data":"f0e85fd662fb176049cabc688cbc9d6192ebef711af93cb71d1cbec771367d03"} Jul 24 22:52:09.909219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:09.909176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" event={"ID":"d335e5cf-3ccf-43e6-887d-b40794ca93b6","Type":"ContainerStarted","Data":"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f"} Jul 24 22:52:09.909721 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:09.909248 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:09.926340 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:09.926260 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" podStartSLOduration=2.637067111 podStartE2EDuration="2.926240505s" totalImagesPullingTime="289.173394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:09.925806251 +0000 UTC m=+2237.288245573" watchObservedRunningTime="2026-07-24 22:52:09.926240505 +0000 UTC m=+2237.288679826" Jul 24 22:52:11.749596 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:11.749566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:15.919974 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:15.919891 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:52:23.602655 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:23.602597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:52:53.460168 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:52:53.460134 2576 scope.go:117] "RemoveContainer" containerID="23684e169878918651de508ddfb575b7a5a2bb84a350ff49976ed5716e0d96bf" Jul 24 22:53:00.143662 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:00.143617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:41.426900 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:41.426864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:42.352458 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:42.352416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:52.787414 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:52.787332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:53:52.787808 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:52.787553 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" podUID="7ae33444-5b8c-4053-8fda-0662839cc808" containerName="maas-api" containerID="cri-o://121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922" gracePeriod=30 Jul 24 22:53:52.806610 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:52.806569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:52.869329 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:52.869272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:52.929738 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:52.929683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:53.058783 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.058753 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:53:53.113148 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.113115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkmhm\" (UniqueName: \"kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm\") pod \"7ae33444-5b8c-4053-8fda-0662839cc808\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " Jul 24 22:53:53.113148 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.113150 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls\") pod \"7ae33444-5b8c-4053-8fda-0662839cc808\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " Jul 24 22:53:53.113461 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.113314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca\") pod \"7ae33444-5b8c-4053-8fda-0662839cc808\" (UID: \"7ae33444-5b8c-4053-8fda-0662839cc808\") " Jul 24 22:53:53.113709 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.113682 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ae33444-5b8c-4053-8fda-0662839cc808" (UID: "7ae33444-5b8c-4053-8fda-0662839cc808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:53:53.115504 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.115475 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ae33444-5b8c-4053-8fda-0662839cc808" (UID: "7ae33444-5b8c-4053-8fda-0662839cc808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:53:53.115606 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.115532 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm" (OuterVolumeSpecName: "kube-api-access-lkmhm") pod "7ae33444-5b8c-4053-8fda-0662839cc808" (UID: "7ae33444-5b8c-4053-8fda-0662839cc808"). InnerVolumeSpecName "kube-api-access-lkmhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:53:53.214124 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.214089 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae33444-5b8c-4053-8fda-0662839cc808-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:53:53.214124 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.214117 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmhm\" (UniqueName: \"kubernetes.io/projected/7ae33444-5b8c-4053-8fda-0662839cc808-kube-api-access-lkmhm\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:53:53.214124 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.214127 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae33444-5b8c-4053-8fda-0662839cc808-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:53:53.499215 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.499181 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ae33444-5b8c-4053-8fda-0662839cc808" containerID="121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922" exitCode=0 Jul 24 22:53:53.499415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.499240 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" Jul 24 22:53:53.499415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.499262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" event={"ID":"7ae33444-5b8c-4053-8fda-0662839cc808","Type":"ContainerDied","Data":"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922"} Jul 24 22:53:53.499415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.499350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w" event={"ID":"7ae33444-5b8c-4053-8fda-0662839cc808","Type":"ContainerDied","Data":"f07d9bc823257413e0a248381055af073b01d5fed7d36f3fe5061412dced1941"} Jul 24 22:53:53.499415 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.499377 2576 scope.go:117] "RemoveContainer" containerID="121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922" Jul 24 22:53:53.509287 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.509263 2576 scope.go:117] "RemoveContainer" containerID="121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922" Jul 24 22:53:53.509609 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:53:53.509585 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922\": container with ID starting with 121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922 not found: ID does not exist" containerID="121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922" Jul 24 22:53:53.509721 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.509615 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922"} err="failed to get container status \"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922\": rpc error: code = NotFound desc = could not find container \"121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922\": container with ID starting with 121ac5fc6ab542419b27b66f97432f4ab73aa8cd67014ac3c0507d13b2381922 not found: ID does not exist" Jul 24 22:53:53.515778 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.515750 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:53:53.519279 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:53.519254 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11557e-84c9c8dd4c-dxs9w"] Jul 24 22:53:55.182884 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:55.182852 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae33444-5b8c-4053-8fda-0662839cc808" path="/var/lib/kubelet/pods/7ae33444-5b8c-4053-8fda-0662839cc808/volumes" Jul 24 22:53:57.516732 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:57.516695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:53:57.971648 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:53:57.971610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:03.661464 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:03.661415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:03.762008 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:03.761960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:03.916823 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:03.916729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:04.203654 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.203622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:04.261707 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.261670 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj"] Jul 24 22:54:04.262119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.262105 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae33444-5b8c-4053-8fda-0662839cc808" containerName="maas-api" Jul 24 22:54:04.262119 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.262121 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae33444-5b8c-4053-8fda-0662839cc808" containerName="maas-api" Jul 24 22:54:04.262219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.262209 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae33444-5b8c-4053-8fda-0662839cc808" containerName="maas-api" Jul 24 22:54:04.265276 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.265259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:04.267730 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.267708 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:54:04.273503 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.273473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj"] Jul 24 22:54:04.319863 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.319820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj\" (UID: \"3fb364dd-702e-4bde-9014-a3ef1afa7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:04.420724 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.420684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj\" (UID: \"3fb364dd-702e-4bde-9014-a3ef1afa7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:04.421322 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.421273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj\" (UID: \"3fb364dd-702e-4bde-9014-a3ef1afa7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:04.577165 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.577048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:04.725817 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:04.725791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj"] Jul 24 22:54:04.728726 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:54:04.728696 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb364dd_702e_4bde_9014_a3ef1afa7dd6.slice/crio-48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72 WatchSource:0}: Error finding container 48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72: Status 404 returned error can't find the container with id 48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72 Jul 24 22:54:05.569495 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:05.569465 2576 generic.go:356] "Generic (PLEG): container finished" podID="3fb364dd-702e-4bde-9014-a3ef1afa7dd6" containerID="7f4d6a02f13cf20baba3d2b5b2e0c4751d53cb617ea5dce52c2ee471ce031a9e" exitCode=0 Jul 24 22:54:05.569680 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:05.569561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" event={"ID":"3fb364dd-702e-4bde-9014-a3ef1afa7dd6","Type":"ContainerDied","Data":"7f4d6a02f13cf20baba3d2b5b2e0c4751d53cb617ea5dce52c2ee471ce031a9e"} Jul 24 22:54:05.569680 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:05.569611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" event={"ID":"3fb364dd-702e-4bde-9014-a3ef1afa7dd6","Type":"ContainerStarted","Data":"48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72"} Jul 24 22:54:06.722057 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:06.722025 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:06.843633 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:06.843532 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca\") pod \"3fb364dd-702e-4bde-9014-a3ef1afa7dd6\" (UID: \"3fb364dd-702e-4bde-9014-a3ef1afa7dd6\") " Jul 24 22:54:06.843896 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:06.843874 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fb364dd-702e-4bde-9014-a3ef1afa7dd6" (UID: "3fb364dd-702e-4bde-9014-a3ef1afa7dd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:54:06.944692 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:06.944655 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb364dd-702e-4bde-9014-a3ef1afa7dd6-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:54:07.581898 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:07.581866 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" Jul 24 22:54:07.582079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:07.581905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj" event={"ID":"3fb364dd-702e-4bde-9014-a3ef1afa7dd6","Type":"ContainerDied","Data":"48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72"} Jul 24 22:54:07.582079 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:07.581955 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d4667b7a974565a7cdcdf0858b96f01763cf7038fc9f8407f51925de90dc72" Jul 24 22:54:14.608383 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.608284 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:54:14.609050 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.608664 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" podUID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" containerName="maas-api" containerID="cri-o://e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f" gracePeriod=30 Jul 24 22:54:14.636977 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.636930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:14.709768 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.709723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:14.811879 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.811843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:14.869702 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.869632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:54:14.921478 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.921450 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") pod \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " Jul 24 22:54:14.921654 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.921569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmsf6\" (UniqueName: \"kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6\") pod \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " Jul 24 22:54:14.921654 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.921628 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca\") pod \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\" (UID: \"d335e5cf-3ccf-43e6-887d-b40794ca93b6\") " Jul 24 22:54:14.921950 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.921920 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d335e5cf-3ccf-43e6-887d-b40794ca93b6" (UID: "d335e5cf-3ccf-43e6-887d-b40794ca93b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:54:14.923825 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.923797 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d335e5cf-3ccf-43e6-887d-b40794ca93b6" (UID: "d335e5cf-3ccf-43e6-887d-b40794ca93b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:54:14.923935 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:14.923839 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6" (OuterVolumeSpecName: "kube-api-access-bmsf6") pod "d335e5cf-3ccf-43e6-887d-b40794ca93b6" (UID: "d335e5cf-3ccf-43e6-887d-b40794ca93b6"). InnerVolumeSpecName "kube-api-access-bmsf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:54:15.023178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.023137 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmsf6\" (UniqueName: \"kubernetes.io/projected/d335e5cf-3ccf-43e6-887d-b40794ca93b6-kube-api-access-bmsf6\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:54:15.023178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.023174 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d335e5cf-3ccf-43e6-887d-b40794ca93b6-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:54:15.023178 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.023189 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d335e5cf-3ccf-43e6-887d-b40794ca93b6-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:54:15.630463 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.630428 2576 generic.go:356] "Generic (PLEG): container finished" podID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" containerID="e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f" exitCode=0 Jul 24 22:54:15.630965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.630476 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" Jul 24 22:54:15.630965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.630514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" event={"ID":"d335e5cf-3ccf-43e6-887d-b40794ca93b6","Type":"ContainerDied","Data":"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f"} Jul 24 22:54:15.630965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.630589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t" event={"ID":"d335e5cf-3ccf-43e6-887d-b40794ca93b6","Type":"ContainerDied","Data":"f0e85fd662fb176049cabc688cbc9d6192ebef711af93cb71d1cbec771367d03"} Jul 24 22:54:15.630965 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.630625 2576 scope.go:117] "RemoveContainer" containerID="e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f" Jul 24 22:54:15.640257 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.640217 2576 scope.go:117] "RemoveContainer" containerID="e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f" Jul 24 22:54:15.640602 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:54:15.640576 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f\": container with ID starting with e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f not found: ID does not exist" containerID="e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f" Jul 24 22:54:15.640657 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.640617 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f"} err="failed to get container status \"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f\": rpc error: code = NotFound desc = could not find container \"e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f\": container with ID starting with e9ddbb289c0a7e73e028b5ea74b0ccc2617884c4c6b956618582c5e3071dd98f not found: ID does not exist" Jul 24 22:54:15.647219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.647194 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:54:15.649219 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:15.649190 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-422c4e-5fd87b568b-f4k8t"] Jul 24 22:54:17.182787 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:17.182731 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" path="/var/lib/kubelet/pods/d335e5cf-3ccf-43e6-887d-b40794ca93b6/volumes" Jul 24 22:54:19.339512 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:19.339463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:19.340877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:19.340853 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj"] Jul 24 22:54:19.342936 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:19.342917 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-422c4e-mhvxj"] Jul 24 22:54:19.711366 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:19.711332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:21.182545 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:21.182511 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb364dd-702e-4bde-9014-a3ef1afa7dd6" path="/var/lib/kubelet/pods/3fb364dd-702e-4bde-9014-a3ef1afa7dd6/volumes" Jul 24 22:54:25.807207 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:25.807163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:25.910450 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:25.910410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:26.073332 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:26.073200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:26.292229 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:26.292192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:26.631883 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:26.631845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:26.955935 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:26.955879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:27.136436 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:27.136390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:30.201927 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.201885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:30.291835 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.291775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:30.422750 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.422719 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:54:30.423135 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423123 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" containerName="maas-api" Jul 24 22:54:30.423184 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423137 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" containerName="maas-api" Jul 24 22:54:30.423184 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423146 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb364dd-702e-4bde-9014-a3ef1afa7dd6" containerName="revoke-keys" Jul 24 22:54:30.423184 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423152 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb364dd-702e-4bde-9014-a3ef1afa7dd6" containerName="revoke-keys" Jul 24 22:54:30.423277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423238 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d335e5cf-3ccf-43e6-887d-b40794ca93b6" containerName="maas-api" Jul 24 22:54:30.423277 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.423246 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb364dd-702e-4bde-9014-a3ef1afa7dd6" containerName="revoke-keys" Jul 24 22:54:30.426272 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.426256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.432439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.432413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aac4b8-serving-cert\"" Jul 24 22:54:30.432439 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.432431 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:54:30.438162 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.438131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:54:30.460985 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.460889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:30.581223 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.581182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.581223 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.581227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.581484 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.581389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzlq\" (UniqueName: \"kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.682758 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.682717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.682758 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.682764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.683011 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.682845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzlq\" (UniqueName: \"kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.683011 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:54:30.682964 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-serving-cert: secret "maas-api-e2e-api-a-aac4b8-serving-cert" not found Jul 24 22:54:30.683093 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:54:30.683038 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls podName:ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8 nodeName:}" failed. No retries permitted until 2026-07-24 22:54:31.183016493 +0000 UTC m=+2378.545455800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls") pod "maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" (UID: "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8") : secret "maas-api-e2e-api-a-aac4b8-serving-cert" not found Jul 24 22:54:30.683465 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.683445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.692779 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.692752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzlq\" (UniqueName: \"kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:30.811235 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.811136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:30.934540 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:30.934500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:31.037049 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.037009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:31.189776 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.189678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:31.192486 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.192451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") pod \"maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:31.337624 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.337592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:31.700404 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.700374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:54:31.702629 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:54:31.702602 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2ec384_9eb0_44c4_a6db_f4831bb8ecd8.slice/crio-ecce11a26630960072c093cbe123a5e381a1277e503cc31b0788beef1d74e5e7 WatchSource:0}: Error finding container ecce11a26630960072c093cbe123a5e381a1277e503cc31b0788beef1d74e5e7: Status 404 returned error can't find the container with id ecce11a26630960072c093cbe123a5e381a1277e503cc31b0788beef1d74e5e7 Jul 24 22:54:31.704601 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.704582 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:54:31.723270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:31.723226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" event={"ID":"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8","Type":"ContainerStarted","Data":"ecce11a26630960072c093cbe123a5e381a1277e503cc31b0788beef1d74e5e7"} Jul 24 22:54:32.731159 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:32.731124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" event={"ID":"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8","Type":"ContainerStarted","Data":"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b"} Jul 24 22:54:32.731627 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:32.731189 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:32.749062 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:32.749006 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" podStartSLOduration=2.286844841 podStartE2EDuration="2.748990709s" totalImagesPullingTime="462.145868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:32.74783809 +0000 UTC m=+2380.110277425" watchObservedRunningTime="2026-07-24 22:54:32.748990709 +0000 UTC m=+2380.111430029" Jul 24 22:54:34.570348 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:34.570275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:35.025952 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:35.025913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:35.337743 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:35.337656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:35.454320 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:35.454266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.160497 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.160454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.224949 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.224909 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:54:38.228696 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.228673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.231264 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.231230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-76c851-serving-cert\"" Jul 24 22:54:38.243064 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.243039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:54:38.257718 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.257664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.257957 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.257930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.258080 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.258062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.258131 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.258101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbpb\" (UniqueName: \"kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.323101 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.323065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.359454 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.359414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.359643 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.359497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.359643 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.359533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbpb\" (UniqueName: \"kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.360315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.360229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.362315 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.362228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.368407 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.368385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbpb\" (UniqueName: \"kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb\") pod \"maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.423529 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.423434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.541023 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.540989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:38.707698 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.707669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:54:38.710053 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:54:38.710024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9936d217_d87e_4a4e_8d89_12d1eb8fc595.slice/crio-46542c3a87a9bb4d99fdd2ef1554c268b122302a420c554f1219da897d8b2630 WatchSource:0}: Error finding container 46542c3a87a9bb4d99fdd2ef1554c268b122302a420c554f1219da897d8b2630: Status 404 returned error can't find the container with id 46542c3a87a9bb4d99fdd2ef1554c268b122302a420c554f1219da897d8b2630 Jul 24 22:54:38.742164 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.742133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:54:38.768602 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.768564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" event={"ID":"9936d217-d87e-4a4e-8d89-12d1eb8fc595","Type":"ContainerStarted","Data":"46542c3a87a9bb4d99fdd2ef1554c268b122302a420c554f1219da897d8b2630"} Jul 24 22:54:38.832113 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.832081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:38.923802 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:38.923752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:39.776449 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:39.776410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" event={"ID":"9936d217-d87e-4a4e-8d89-12d1eb8fc595","Type":"ContainerStarted","Data":"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924"} Jul 24 22:54:39.776890 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:39.776514 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:39.794082 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:39.794020 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" podStartSLOduration=1.325592083 podStartE2EDuration="1.793995511s" totalImagesPullingTime="468.403428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:39.793854709 +0000 UTC m=+2387.156294032" watchObservedRunningTime="2026-07-24 22:54:39.793995511 +0000 UTC m=+2387.156434832" Jul 24 22:54:42.606256 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:42.606184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:54:45.789895 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:45.789865 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:54:53.260664 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:53.260640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:54:53.267136 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:53.267102 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:54:53.268380 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:53.268358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:54:53.274059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:53.274033 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:54:53.490195 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:54:53.490155 2576 scope.go:117] "RemoveContainer" containerID="83afe36852bb57a0a3dbc1da3b86cf7bd0ec4224b7b1226daa53cce0ffaf94bb" Jul 24 22:55:05.652121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.652067 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:55:05.652658 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.652495 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" podUID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" containerName="maas-api" containerID="cri-o://90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b" gracePeriod=30 Jul 24 22:55:05.682674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.682636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:05.752678 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.752634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:05.817224 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.817183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:05.920041 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.919976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:55:05.932431 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.932402 2576 generic.go:356] "Generic (PLEG): container finished" podID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" containerID="90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b" exitCode=0 Jul 24 22:55:05.932614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.932450 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" Jul 24 22:55:05.932614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.932491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" event={"ID":"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8","Type":"ContainerDied","Data":"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b"} Jul 24 22:55:05.932614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.932544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l" event={"ID":"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8","Type":"ContainerDied","Data":"ecce11a26630960072c093cbe123a5e381a1277e503cc31b0788beef1d74e5e7"} Jul 24 22:55:05.932614 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.932568 2576 scope.go:117] "RemoveContainer" containerID="90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b" Jul 24 22:55:05.944535 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.944493 2576 scope.go:117] "RemoveContainer" containerID="90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b" Jul 24 22:55:05.944901 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:05.944877 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b\": container with ID starting with 90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b not found: ID does not exist" containerID="90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b" Jul 24 22:55:05.945019 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:05.944908 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b"} err="failed to get container status \"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b\": rpc error: code = NotFound desc = could not find container \"90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b\": container with ID starting with 90086013efab815c6bf8ed300fae1df9593a214cb012298f7f06a0603e1c836b not found: ID does not exist" Jul 24 22:55:06.032733 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.032697 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrzlq\" (UniqueName: \"kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq\") pod \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " Jul 24 22:55:06.032938 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.032745 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca\") pod \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " Jul 24 22:55:06.032938 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.032850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") pod \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\" (UID: \"ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8\") " Jul 24 22:55:06.033186 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.033157 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" (UID: "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:55:06.035137 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.035109 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" (UID: "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:55:06.035270 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.035194 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq" (OuterVolumeSpecName: "kube-api-access-jrzlq") pod "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" (UID: "ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8"). InnerVolumeSpecName "kube-api-access-jrzlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:55:06.133833 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.133789 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzlq\" (UniqueName: \"kubernetes.io/projected/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-kube-api-access-jrzlq\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:06.133833 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.133828 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:06.133833 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.133838 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:06.254653 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.254619 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:55:06.258374 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:06.258321 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac4b8-6dcfc445c7-vmj4l"] Jul 24 22:55:07.182037 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:07.181999 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" path="/var/lib/kubelet/pods/ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8/volumes" Jul 24 22:55:10.554494 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:10.554456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:10.924021 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:10.923925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:16.872359 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:16.872269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:16.975192 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:16.975152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:17.135430 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.135335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:17.352090 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.352047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:17.474472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.474441 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw"] Jul 24 22:55:17.474890 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.474876 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" containerName="maas-api" Jul 24 22:55:17.474936 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.474894 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" containerName="maas-api" Jul 24 22:55:17.474982 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.474972 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2ec384-9eb0-44c4-a6db-f4831bb8ecd8" containerName="maas-api" Jul 24 22:55:17.479339 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.479322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:17.482082 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.482054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 22:55:17.485575 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.485554 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw"] Jul 24 22:55:17.546794 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.546758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76c851-rschw\" (UID: \"a1553c3a-f03c-4452-aa2a-f1311d068836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:17.648033 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.647998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76c851-rschw\" (UID: \"a1553c3a-f03c-4452-aa2a-f1311d068836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:17.648650 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.648634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76c851-rschw\" (UID: \"a1553c3a-f03c-4452-aa2a-f1311d068836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:17.791379 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.791274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:17.939093 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:17.939064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw"] Jul 24 22:55:17.941287 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:55:17.941255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1553c3a_f03c_4452_aa2a_f1311d068836.slice/crio-0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8 WatchSource:0}: Error finding container 0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8: Status 404 returned error can't find the container with id 0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8 Jul 24 22:55:18.005863 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:18.005828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" event={"ID":"a1553c3a-f03c-4452-aa2a-f1311d068836","Type":"ContainerStarted","Data":"0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8"} Jul 24 22:55:19.013026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:19.012925 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1553c3a-f03c-4452-aa2a-f1311d068836" containerID="929e66de94478b5c210c4888592f36572ba5fa03d4b9a15ce5f16c08c239163a" exitCode=0 Jul 24 22:55:19.013026 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:19.012975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" event={"ID":"a1553c3a-f03c-4452-aa2a-f1311d068836","Type":"ContainerDied","Data":"929e66de94478b5c210c4888592f36572ba5fa03d4b9a15ce5f16c08c239163a"} Jul 24 22:55:20.162244 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:20.162217 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:20.270472 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:20.270379 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca\") pod \"a1553c3a-f03c-4452-aa2a-f1311d068836\" (UID: \"a1553c3a-f03c-4452-aa2a-f1311d068836\") " Jul 24 22:55:20.270769 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:20.270746 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1553c3a-f03c-4452-aa2a-f1311d068836" (UID: "a1553c3a-f03c-4452-aa2a-f1311d068836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:55:20.371014 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:20.370982 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1553c3a-f03c-4452-aa2a-f1311d068836-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:21.025123 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:21.025090 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" Jul 24 22:55:21.025123 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:21.025120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw" event={"ID":"a1553c3a-f03c-4452-aa2a-f1311d068836","Type":"ContainerDied","Data":"0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8"} Jul 24 22:55:21.025358 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:21.025168 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0748d83926a86cba9ac67b17bf989b4d8da608baee0d6f01f86defdec3c1fbe8" Jul 24 22:55:32.989899 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:32.989860 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:55:32.990491 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:32.990093 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" podUID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" containerName="maas-api" containerID="cri-o://efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924" gracePeriod=30 Jul 24 22:55:33.011284 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.011257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:33.076073 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.076043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:33.177554 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.177516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:33.242814 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.242748 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:55:33.291735 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.291685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbpb\" (UniqueName: \"kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb\") pod \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " Jul 24 22:55:33.291892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.291765 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca\") pod \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " Jul 24 22:55:33.291892 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.291866 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls\") pod \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\" (UID: \"9936d217-d87e-4a4e-8d89-12d1eb8fc595\") " Jul 24 22:55:33.292091 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.292070 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9936d217-d87e-4a4e-8d89-12d1eb8fc595" (UID: "9936d217-d87e-4a4e-8d89-12d1eb8fc595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:55:33.292261 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.292248 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9936d217-d87e-4a4e-8d89-12d1eb8fc595-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:33.294039 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.294015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9936d217-d87e-4a4e-8d89-12d1eb8fc595" (UID: "9936d217-d87e-4a4e-8d89-12d1eb8fc595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:55:33.294114 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.294061 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb" (OuterVolumeSpecName: "kube-api-access-sgbpb") pod "9936d217-d87e-4a4e-8d89-12d1eb8fc595" (UID: "9936d217-d87e-4a4e-8d89-12d1eb8fc595"). InnerVolumeSpecName "kube-api-access-sgbpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:55:33.393546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.393512 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9936d217-d87e-4a4e-8d89-12d1eb8fc595-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:33.393546 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:33.393543 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbpb\" (UniqueName: \"kubernetes.io/projected/9936d217-d87e-4a4e-8d89-12d1eb8fc595-kube-api-access-sgbpb\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 22:55:34.103720 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.103677 2576 generic.go:356] "Generic (PLEG): container finished" podID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" containerID="efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924" exitCode=0 Jul 24 22:55:34.104236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.103740 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" Jul 24 22:55:34.104236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.103771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" event={"ID":"9936d217-d87e-4a4e-8d89-12d1eb8fc595","Type":"ContainerDied","Data":"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924"} Jul 24 22:55:34.104236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.103865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627" event={"ID":"9936d217-d87e-4a4e-8d89-12d1eb8fc595","Type":"ContainerDied","Data":"46542c3a87a9bb4d99fdd2ef1554c268b122302a420c554f1219da897d8b2630"} Jul 24 22:55:34.104236 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.103896 2576 scope.go:117] "RemoveContainer" containerID="efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924" Jul 24 22:55:34.113492 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.113476 2576 scope.go:117] "RemoveContainer" containerID="efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924" Jul 24 22:55:34.113771 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:34.113750 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924\": container with ID starting with efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924 not found: ID does not exist" containerID="efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924" Jul 24 22:55:34.113832 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.113784 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924"} err="failed to get container status \"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924\": rpc error: code = NotFound desc = could not find container \"efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924\": container with ID starting with efec78d19cd22d5740253ac6a062e8904af5259f89cf20ee4e7bd02825d84924 not found: ID does not exist" Jul 24 22:55:34.128176 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.128148 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:55:34.131437 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:34.131414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76c851-6c4c8d8b65-2l627"] Jul 24 22:55:35.182462 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:35.182433 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" path="/var/lib/kubelet/pods/9936d217-d87e-4a4e-8d89-12d1eb8fc595/volumes" Jul 24 22:55:37.536770 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:37.536736 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw"] Jul 24 22:55:37.538491 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:37.538447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:37.538959 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:37.538938 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76c851-rschw"] Jul 24 22:55:37.960399 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:37.960350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:39.182024 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:39.181991 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1553c3a-f03c-4452-aa2a-f1311d068836" path="/var/lib/kubelet/pods/a1553c3a-f03c-4452-aa2a-f1311d068836/volumes" Jul 24 22:55:43.705478 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:43.705444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:43.806792 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:43.806758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:43.954831 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:43.954799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:44.165906 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:44.165795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:44.498193 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:44.498154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:44.801458 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:44.801372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:45.147030 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:45.146942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:47.554827 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.554784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:47.652674 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.652639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:47.718782 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.718750 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 22:55:47.719541 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719514 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1553c3a-f03c-4452-aa2a-f1311d068836" containerName="revoke-keys" Jul 24 22:55:47.719541 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719540 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1553c3a-f03c-4452-aa2a-f1311d068836" containerName="revoke-keys" Jul 24 22:55:47.719728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719563 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" containerName="maas-api" Jul 24 22:55:47.719728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719572 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" containerName="maas-api" Jul 24 22:55:47.719728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719676 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1553c3a-f03c-4452-aa2a-f1311d068836" containerName="revoke-keys" Jul 24 22:55:47.719728 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.719690 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9936d217-d87e-4a4e-8d89-12d1eb8fc595" containerName="maas-api" Jul 24 22:55:47.723944 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.723928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.726566 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.726518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gqpg7\"" Jul 24 22:55:47.726690 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.726652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-499dd1-serving-cert\"" Jul 24 22:55:47.733085 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.733059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 22:55:47.748889 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.748861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:47.834635 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.834537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q4z\" (UniqueName: \"kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.834635 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.834584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.834839 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.834698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.935909 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.935872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.936094 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.935988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q4z\" (UniqueName: \"kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.936094 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.936027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.936094 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:47.936037 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-serving-cert: secret "maas-api-e2e-shared-a-499dd1-serving-cert" not found Jul 24 22:55:47.936206 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:47.936119 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls podName:5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1 nodeName:}" failed. No retries permitted until 2026-07-24 22:55:48.436095001 +0000 UTC m=+2455.798534303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls") pod "maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" (UID: "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1") : secret "maas-api-e2e-shared-a-499dd1-serving-cert" not found Jul 24 22:55:47.936708 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.936690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:47.945517 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:47.945482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q4z\" (UniqueName: \"kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:48.208066 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.208025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:48.303973 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.303929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:48.446173 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.446130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:48.448773 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.448743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") pod \"maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:48.636340 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.636233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:48.785890 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:48.785864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 22:55:48.788015 ip-10-0-139-136 kubenswrapper[2576]: W0724 22:55:48.787985 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1bc44_8862_4e0f_a297_6ddcbcc36aa1.slice/crio-3a1655d5ecb156ae32b78d40d192aab72180a1b0547f68adee84b83e2543690a WatchSource:0}: Error finding container 3a1655d5ecb156ae32b78d40d192aab72180a1b0547f68adee84b83e2543690a: Status 404 returned error can't find the container with id 3a1655d5ecb156ae32b78d40d192aab72180a1b0547f68adee84b83e2543690a Jul 24 22:55:49.200153 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:49.200120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" event={"ID":"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1","Type":"ContainerStarted","Data":"3a1655d5ecb156ae32b78d40d192aab72180a1b0547f68adee84b83e2543690a"} Jul 24 22:55:50.207368 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:50.207331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" event={"ID":"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1","Type":"ContainerStarted","Data":"a8117ff69156a905f55118baf25ef64e01b99d7fa8401afee0c16d01d49641cd"} Jul 24 22:55:50.207790 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:50.207400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:50.223986 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:50.223934 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" podStartSLOduration=2.662231146 podStartE2EDuration="3.22390994s" totalImagesPullingTime="561.678794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:50.222478719 +0000 UTC m=+2457.584918066" watchObservedRunningTime="2026-07-24 22:55:50.22390994 +0000 UTC m=+2457.586349261" Jul 24 22:55:52.053055 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:52.053010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:52.367059 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:52.366967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:52.689976 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:52.689882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:53.143580 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:53.143542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:53.523805 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:53.523773 2576 scope.go:117] "RemoveContainer" containerID="eccb1f271738a37a2f4734dc59b9d44a969a7dc3e05dcdd8a2889d9026fc5b32" Jul 24 22:55:55.584574 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.584526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:55.666544 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.666501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:55.667533 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.667510 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 22:55:55.672732 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.672713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.675475 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.675453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e15986-serving-cert\"" Jul 24 22:55:55.683398 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.683372 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 22:55:55.718759 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.718718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djbh\" (UniqueName: \"kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.718942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.718773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.718942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.718800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.740188 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.740152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:55.819694 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.819657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djbh\" (UniqueName: \"kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.819694 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.819699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.820001 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.819724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.820001 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:55.819907 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-serving-cert: secret "maas-api-e2e-shared-b-e15986-serving-cert" not found Jul 24 22:55:55.820001 ip-10-0-139-136 kubenswrapper[2576]: E0724 22:55:55.819986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls podName:56da2490-0f84-4d74-8b05-f0d783d299be nodeName:}" failed. No retries permitted until 2026-07-24 22:55:56.319966727 +0000 UTC m=+2463.682406031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls") pod "maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" (UID: "56da2490-0f84-4d74-8b05-f0d783d299be") : secret "maas-api-e2e-shared-b-e15986-serving-cert" not found Jul 24 22:55:55.820512 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.820475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.836628 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.836559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djbh\" (UniqueName: \"kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:55.914237 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:55.914205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:56.221591 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.221563 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 22:55:56.251442 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.251352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:56.325038 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.324997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:56.327632 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.327610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") pod \"maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:56.350877 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.350829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:55:56.585228 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.585140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:56.735382 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:56.735347 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 22:55:57.247942 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:57.247893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" event={"ID":"56da2490-0f84-4d74-8b05-f0d783d299be","Type":"ContainerStarted","Data":"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75"} Jul 24 22:55:57.248116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:57.247965 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:55:57.248116 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:57.247984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" event={"ID":"56da2490-0f84-4d74-8b05-f0d783d299be","Type":"ContainerStarted","Data":"d06dc3fe6040a7f23103a511c0d940c0b574329606843512a1cf8fabdb922633"} Jul 24 22:55:57.267056 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:55:57.266987 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" podStartSLOduration=1.879641286 podStartE2EDuration="2.266970955s" totalImagesPullingTime="387.329669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:57.265278672 +0000 UTC m=+2464.627717993" watchObservedRunningTime="2026-07-24 22:55:57.266970955 +0000 UTC m=+2464.629410276" Jul 24 22:56:00.062005 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:00.061969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:56:01.631221 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:01.631188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:56:03.259995 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:03.259959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 22:56:08.031694 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:08.031653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:56:27.326693 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:27.326656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:56:33.229427 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:56:33.229389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:57:52.848072 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:57:52.848027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:58:03.564118 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:58:03.564067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:58:04.714599 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:58:04.714560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:58:32.876954 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:58:32.876875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:58:33.709102 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:58:33.709061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:58:45.734578 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:58:45.734532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:59:40.242519 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:59:40.242480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 22:59:53.295959 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:59:53.295877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:59:53.303916 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:59:53.303889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 22:59:53.304121 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:59:53.304060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 22:59:53.311665 ip-10-0-139-136 kubenswrapper[2576]: I0724 22:59:53.311640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 23:00:00.133558 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.133513 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr"] Jul 24 23:00:00.137251 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.137229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" Jul 24 23:00:00.140097 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.140072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:00:00.149311 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.149255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr"] Jul 24 23:00:00.150667 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.150642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" Jul 24 23:00:00.299830 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.299799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr"] Jul 24 23:00:00.302626 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:00:00.302584 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dedbfdf_067a_43f3_a56c_ed5285c9611a.slice/crio-35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65 WatchSource:0}: Error finding container 35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65: Status 404 returned error can't find the container with id 35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65 Jul 24 23:00:00.606151 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:00.606105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" event={"ID":"3dedbfdf-067a-43f3-a56c-ed5285c9611a","Type":"ContainerStarted","Data":"35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65"} Jul 24 23:00:01.613469 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:01.613361 2576 generic.go:356] "Generic (PLEG): container finished" podID="3dedbfdf-067a-43f3-a56c-ed5285c9611a" containerID="41180a89c4de60ef38041ae3ba752bd2c45ec02b43fe650d0fb53edcfe5b6fb5" exitCode=0 Jul 24 23:00:01.613469 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:01.613423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" event={"ID":"3dedbfdf-067a-43f3-a56c-ed5285c9611a","Type":"ContainerDied","Data":"41180a89c4de60ef38041ae3ba752bd2c45ec02b43fe650d0fb53edcfe5b6fb5"} Jul 24 23:00:02.775032 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:02.774999 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" Jul 24 23:00:03.627715 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:03.627613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" event={"ID":"3dedbfdf-067a-43f3-a56c-ed5285c9611a","Type":"ContainerDied","Data":"35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65"} Jul 24 23:00:03.627715 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:03.627669 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c57c548d04d92f65e7386f029cd6050712914625824daf6d4cff2de2c94a65" Jul 24 23:00:03.627715 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:03.627631 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7x7rr" Jul 24 23:00:28.310271 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:28.310222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:00:30.418138 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:30.418105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:00:31.213533 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:31.213493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:00:42.537044 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:42.537002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:00:53.586060 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:00:53.585952 2576 scope.go:117] "RemoveContainer" containerID="7f4d6a02f13cf20baba3d2b5b2e0c4751d53cb617ea5dce52c2ee471ce031a9e" Jul 24 23:01:03.649797 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:03.649748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:01:29.855774 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:29.855680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:01:48.997569 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:48.997531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:01:49.808472 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:49.808437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:01:50.706421 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.706383 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825558059696"] Jul 24 23:01:50.706866 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.706850 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dedbfdf-067a-43f3-a56c-ed5285c9611a" containerName="cleanup" Jul 24 23:01:50.706912 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.706867 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dedbfdf-067a-43f3-a56c-ed5285c9611a" containerName="cleanup" Jul 24 23:01:50.706965 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.706956 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dedbfdf-067a-43f3-a56c-ed5285c9611a" containerName="cleanup" Jul 24 23:01:50.710199 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.710180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:50.713348 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.713116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vq9qt\"" Jul 24 23:01:50.716520 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.716496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825558059696"] Jul 24 23:01:50.764625 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.764582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhltq\" (UniqueName: \"kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq\") pod \"test-curl-7537-139825558059696\" (UID: \"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d\") " pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:50.865187 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.865153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhltq\" (UniqueName: \"kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq\") pod \"test-curl-7537-139825558059696\" (UID: \"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d\") " pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:50.873289 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:50.873260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhltq\" (UniqueName: \"kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq\") pod \"test-curl-7537-139825558059696\" (UID: \"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d\") " pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:51.021475 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:51.021381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:51.161809 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:51.161783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825558059696"] Jul 24 23:01:51.163939 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:01:51.163911 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23f10ce2_f2a4_4edb_8fa2_b8cdeb92e61d.slice/crio-8ac689903c8f61fb403334712293d531744a9925df193d4fc3555aa80c56d266 WatchSource:0}: Error finding container 8ac689903c8f61fb403334712293d531744a9925df193d4fc3555aa80c56d266: Status 404 returned error can't find the container with id 8ac689903c8f61fb403334712293d531744a9925df193d4fc3555aa80c56d266 Jul 24 23:01:51.165387 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:51.165354 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:01:51.240735 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:51.240697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825558059696" event={"ID":"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d","Type":"ContainerStarted","Data":"8ac689903c8f61fb403334712293d531744a9925df193d4fc3555aa80c56d266"} Jul 24 23:01:52.247935 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:52.247889 2576 generic.go:356] "Generic (PLEG): container finished" podID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" containerID="74b4b7b80e1837273745da0511a67a04fd3dd1107e709ebbb86eb841e2546313" exitCode=0 Jul 24 23:01:52.248407 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:52.247950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825558059696" event={"ID":"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d","Type":"ContainerDied","Data":"74b4b7b80e1837273745da0511a67a04fd3dd1107e709ebbb86eb841e2546313"} Jul 24 23:01:53.406014 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.405983 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:53.424252 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.424206 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825558059696_23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d/test-curl-7537-139825558059696/0.log" Jul 24 23:01:53.448941 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.448908 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825558059696"] Jul 24 23:01:53.452463 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.452436 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825558059696"] Jul 24 23:01:53.489628 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.489591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhltq\" (UniqueName: \"kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq\") pod \"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d\" (UID: \"23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d\") " Jul 24 23:01:53.491800 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.491771 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq" (OuterVolumeSpecName: "kube-api-access-fhltq") pod "23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" (UID: "23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d"). InnerVolumeSpecName "kube-api-access-fhltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:01:53.562335 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.562220 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825558059824"] Jul 24 23:01:53.562767 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.562752 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" containerName="test-curl-7537-139825558059696" Jul 24 23:01:53.562812 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.562770 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" containerName="test-curl-7537-139825558059696" Jul 24 23:01:53.562851 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.562846 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" containerName="test-curl-7537-139825558059696" Jul 24 23:01:53.566492 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.566469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:53.571571 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.571546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825558059824"] Jul 24 23:01:53.590590 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.590561 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhltq\" (UniqueName: \"kubernetes.io/projected/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d-kube-api-access-fhltq\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:01:53.609794 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.609756 2576 scope.go:117] "RemoveContainer" containerID="929e66de94478b5c210c4888592f36572ba5fa03d4b9a15ce5f16c08c239163a" Jul 24 23:01:53.691502 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.691470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqww7\" (UniqueName: \"kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7\") pod \"test-curl-7537-139825558059824\" (UID: \"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900\") " pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:53.792223 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.792183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqww7\" (UniqueName: \"kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7\") pod \"test-curl-7537-139825558059824\" (UID: \"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900\") " pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:53.801226 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.801192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqww7\" (UniqueName: \"kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7\") pod \"test-curl-7537-139825558059824\" (UID: \"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900\") " pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:53.877966 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:53.877870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:54.032715 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:54.032680 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825558059824"] Jul 24 23:01:54.035256 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:01:54.035228 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f679d4_5e6b_4ad4_94a3_1f68f4d16900.slice/crio-338f46cca01cac9b7a6bfa8352bd8473fb1f3eb4e1bfefd26ec09c49350c0d72 WatchSource:0}: Error finding container 338f46cca01cac9b7a6bfa8352bd8473fb1f3eb4e1bfefd26ec09c49350c0d72: Status 404 returned error can't find the container with id 338f46cca01cac9b7a6bfa8352bd8473fb1f3eb4e1bfefd26ec09c49350c0d72 Jul 24 23:01:54.259835 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:54.259789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825558059824" event={"ID":"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900","Type":"ContainerStarted","Data":"338f46cca01cac9b7a6bfa8352bd8473fb1f3eb4e1bfefd26ec09c49350c0d72"} Jul 24 23:01:54.261145 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:54.261125 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:54.261270 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:54.261129 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac689903c8f61fb403334712293d531744a9925df193d4fc3555aa80c56d266" Jul 24 23:01:54.271611 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:01:54.271584 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-139825558059696\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" pod="opendatahub/test-curl-7537-139825558059696" Jul 24 23:01:55.182991 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:55.182953 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d" path="/var/lib/kubelet/pods/23f10ce2-f2a4-4edb-8fa2-b8cdeb92e61d/volumes" Jul 24 23:01:55.267880 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:55.267848 2576 generic.go:356] "Generic (PLEG): container finished" podID="e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" containerID="e1437f1b6acdc56ed32c30dfc1daed35b675ea6f362ca2f64fe0e071a7fd026f" exitCode=0 Jul 24 23:01:55.268055 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:55.267942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825558059824" event={"ID":"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900","Type":"ContainerDied","Data":"e1437f1b6acdc56ed32c30dfc1daed35b675ea6f362ca2f64fe0e071a7fd026f"} Jul 24 23:01:56.416759 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.416728 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:56.435693 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.435630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825558059824_e3f679d4-5e6b-4ad4-94a3-1f68f4d16900/test-curl-7537-139825558059824/0.log" Jul 24 23:01:56.462268 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.462236 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825558059824"] Jul 24 23:01:56.465119 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.465097 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825558059824"] Jul 24 23:01:56.518351 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.518314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqww7\" (UniqueName: \"kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7\") pod \"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900\" (UID: \"e3f679d4-5e6b-4ad4-94a3-1f68f4d16900\") " Jul 24 23:01:56.520417 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.520391 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7" (OuterVolumeSpecName: "kube-api-access-tqww7") pod "e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" (UID: "e3f679d4-5e6b-4ad4-94a3-1f68f4d16900"). InnerVolumeSpecName "kube-api-access-tqww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:01:56.619100 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.619068 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqww7\" (UniqueName: \"kubernetes.io/projected/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900-kube-api-access-tqww7\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:01:56.622271 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.622238 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:56.622734 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.622716 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" containerName="test-curl-7537-139825558059824" Jul 24 23:01:56.622817 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.622737 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" containerName="test-curl-7537-139825558059824" Jul 24 23:01:56.622893 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.622872 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" containerName="test-curl-7537-139825558059824" Jul 24 23:01:56.626076 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.626057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:56.632415 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.632388 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:56.719796 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.719751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzs8\" (UniqueName: \"kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8\") pod \"test-curl-7537-139825559704752\" (UID: \"7d7f1816-2d88-4052-9d82-39298d30bb41\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:56.821420 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.821323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzs8\" (UniqueName: \"kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8\") pod \"test-curl-7537-139825559704752\" (UID: \"7d7f1816-2d88-4052-9d82-39298d30bb41\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:56.830442 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.830408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzs8\" (UniqueName: \"kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8\") pod \"test-curl-7537-139825559704752\" (UID: \"7d7f1816-2d88-4052-9d82-39298d30bb41\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:56.938421 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:56.938388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:57.085034 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:57.082992 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:57.184807 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:57.184770 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f679d4-5e6b-4ad4-94a3-1f68f4d16900" path="/var/lib/kubelet/pods/e3f679d4-5e6b-4ad4-94a3-1f68f4d16900/volumes" Jul 24 23:01:57.279664 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:57.279634 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825558059824" Jul 24 23:01:57.279664 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:57.279659 2576 scope.go:117] "RemoveContainer" containerID="e1437f1b6acdc56ed32c30dfc1daed35b675ea6f362ca2f64fe0e071a7fd026f" Jul 24 23:01:57.281145 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:57.280957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825559704752" event={"ID":"7d7f1816-2d88-4052-9d82-39298d30bb41","Type":"ContainerStarted","Data":"33cb249591e0cb8e339917bb0b5704df885062105e3704dfdcaebd6cba77013a"} Jul 24 23:01:58.288777 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:58.288745 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d7f1816-2d88-4052-9d82-39298d30bb41" containerID="9140ea4454fec0de093f8b888a9ef4960fd7ae895ffb33fcd6f2fd152ef44c2a" exitCode=0 Jul 24 23:01:58.289173 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:58.288799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825559704752" event={"ID":"7d7f1816-2d88-4052-9d82-39298d30bb41","Type":"ContainerDied","Data":"9140ea4454fec0de093f8b888a9ef4960fd7ae895ffb33fcd6f2fd152ef44c2a"} Jul 24 23:01:59.440126 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.440100 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:59.459522 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.459452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825559704752_7d7f1816-2d88-4052-9d82-39298d30bb41/test-curl-7537-139825559704752/0.log" Jul 24 23:01:59.483556 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.483515 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:59.486803 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.486779 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:59.548683 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.548593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzzs8\" (UniqueName: \"kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8\") pod \"7d7f1816-2d88-4052-9d82-39298d30bb41\" (UID: \"7d7f1816-2d88-4052-9d82-39298d30bb41\") " Jul 24 23:01:59.550701 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.550677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8" (OuterVolumeSpecName: "kube-api-access-rzzs8") pod "7d7f1816-2d88-4052-9d82-39298d30bb41" (UID: "7d7f1816-2d88-4052-9d82-39298d30bb41"). InnerVolumeSpecName "kube-api-access-rzzs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:01:59.603110 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.603077 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:59.603552 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.603538 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7f1816-2d88-4052-9d82-39298d30bb41" containerName="test-curl-7537-139825559704752" Jul 24 23:01:59.603595 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.603553 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7f1816-2d88-4052-9d82-39298d30bb41" containerName="test-curl-7537-139825559704752" Jul 24 23:01:59.603634 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.603629 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7f1816-2d88-4052-9d82-39298d30bb41" containerName="test-curl-7537-139825559704752" Jul 24 23:01:59.606812 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.606791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:59.613326 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.613264 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:01:59.649775 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.649738 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzzs8\" (UniqueName: \"kubernetes.io/projected/7d7f1816-2d88-4052-9d82-39298d30bb41-kube-api-access-rzzs8\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:01:59.750558 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.750524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhrq\" (UniqueName: \"kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq\") pod \"test-curl-7537-139825559704752\" (UID: \"4c7cc22c-51e3-450e-8447-356e21021672\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:59.852009 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.851914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhrq\" (UniqueName: \"kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq\") pod \"test-curl-7537-139825559704752\" (UID: \"4c7cc22c-51e3-450e-8447-356e21021672\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:59.860599 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.860553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhrq\" (UniqueName: \"kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq\") pod \"test-curl-7537-139825559704752\" (UID: \"4c7cc22c-51e3-450e-8447-356e21021672\") " pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:01:59.918171 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:01:59.918136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:02:00.055904 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:00.055868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:02:00.057919 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:02:00.057887 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c7cc22c_51e3_450e_8447_356e21021672.slice/crio-1a061afdea3afd3df442a9832f7360dfdeb7fa5c1c97c3f84b1eae46e1a4beb9 WatchSource:0}: Error finding container 1a061afdea3afd3df442a9832f7360dfdeb7fa5c1c97c3f84b1eae46e1a4beb9: Status 404 returned error can't find the container with id 1a061afdea3afd3df442a9832f7360dfdeb7fa5c1c97c3f84b1eae46e1a4beb9 Jul 24 23:02:00.300400 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:00.300363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825559704752" event={"ID":"4c7cc22c-51e3-450e-8447-356e21021672","Type":"ContainerStarted","Data":"1a061afdea3afd3df442a9832f7360dfdeb7fa5c1c97c3f84b1eae46e1a4beb9"} Jul 24 23:02:00.301725 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:00.301706 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:02:00.301832 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:00.301740 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cb249591e0cb8e339917bb0b5704df885062105e3704dfdcaebd6cba77013a" Jul 24 23:02:00.313632 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:00.313584 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7537-139825559704752" oldPodUID="7d7f1816-2d88-4052-9d82-39298d30bb41" podUID="4c7cc22c-51e3-450e-8447-356e21021672" Jul 24 23:02:01.181986 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:01.181951 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7f1816-2d88-4052-9d82-39298d30bb41" path="/var/lib/kubelet/pods/7d7f1816-2d88-4052-9d82-39298d30bb41/volumes" Jul 24 23:02:01.311987 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:01.311947 2576 generic.go:356] "Generic (PLEG): container finished" podID="4c7cc22c-51e3-450e-8447-356e21021672" containerID="fc1cc3653e184bca50b878eb33fafd3c4b6d4f8e3914e75bb1a4985fbbf029a5" exitCode=0 Jul 24 23:02:01.312165 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:01.312019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825559704752" event={"ID":"4c7cc22c-51e3-450e-8447-356e21021672","Type":"ContainerDied","Data":"fc1cc3653e184bca50b878eb33fafd3c4b6d4f8e3914e75bb1a4985fbbf029a5"} Jul 24 23:02:02.454347 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.454316 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:02:02.475952 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.475887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825559704752_4c7cc22c-51e3-450e-8447-356e21021672/test-curl-7537-139825559704752/0.log" Jul 24 23:02:02.504828 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.504785 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:02:02.508522 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.508496 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825559704752"] Jul 24 23:02:02.581686 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.581594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shhrq\" (UniqueName: \"kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq\") pod \"4c7cc22c-51e3-450e-8447-356e21021672\" (UID: \"4c7cc22c-51e3-450e-8447-356e21021672\") " Jul 24 23:02:02.583905 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.583867 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq" (OuterVolumeSpecName: "kube-api-access-shhrq") pod "4c7cc22c-51e3-450e-8447-356e21021672" (UID: "4c7cc22c-51e3-450e-8447-356e21021672"). InnerVolumeSpecName "kube-api-access-shhrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:02:02.635405 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.633530 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:02.635405 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.634614 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7cc22c-51e3-450e-8447-356e21021672" containerName="test-curl-7537-139825559704752" Jul 24 23:02:02.635405 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.634646 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7cc22c-51e3-450e-8447-356e21021672" containerName="test-curl-7537-139825559704752" Jul 24 23:02:02.635405 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.634872 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7cc22c-51e3-450e-8447-356e21021672" containerName="test-curl-7537-139825559704752" Jul 24 23:02:02.639478 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.639128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:02.644211 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.644174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:02.682956 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.682924 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhrq\" (UniqueName: \"kubernetes.io/projected/4c7cc22c-51e3-450e-8447-356e21021672-kube-api-access-shhrq\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:02:02.784433 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.784396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjpk\" (UniqueName: \"kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk\") pod \"test-curl-7537-139825557984720\" (UID: \"4cb3462b-416c-44b4-9d4c-cc718e93dc02\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:02.885423 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.885321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjpk\" (UniqueName: \"kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk\") pod \"test-curl-7537-139825557984720\" (UID: \"4cb3462b-416c-44b4-9d4c-cc718e93dc02\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:02.893931 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.893897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjpk\" (UniqueName: \"kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk\") pod \"test-curl-7537-139825557984720\" (UID: \"4cb3462b-416c-44b4-9d4c-cc718e93dc02\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:02.950795 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:02.950756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:03.106588 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:03.106545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:03.108190 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:02:03.108162 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb3462b_416c_44b4_9d4c_cc718e93dc02.slice/crio-c70ff23d181a25960091c2c948f399c92ebee47021ee65855ad305c99260dc37 WatchSource:0}: Error finding container c70ff23d181a25960091c2c948f399c92ebee47021ee65855ad305c99260dc37: Status 404 returned error can't find the container with id c70ff23d181a25960091c2c948f399c92ebee47021ee65855ad305c99260dc37 Jul 24 23:02:03.182825 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:03.182793 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7cc22c-51e3-450e-8447-356e21021672" path="/var/lib/kubelet/pods/4c7cc22c-51e3-450e-8447-356e21021672/volumes" Jul 24 23:02:03.324608 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:03.324569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825557984720" event={"ID":"4cb3462b-416c-44b4-9d4c-cc718e93dc02","Type":"ContainerStarted","Data":"c70ff23d181a25960091c2c948f399c92ebee47021ee65855ad305c99260dc37"} Jul 24 23:02:03.325810 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:03.325795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825559704752" Jul 24 23:02:03.325923 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:03.325833 2576 scope.go:117] "RemoveContainer" containerID="fc1cc3653e184bca50b878eb33fafd3c4b6d4f8e3914e75bb1a4985fbbf029a5" Jul 24 23:02:04.333802 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:04.333769 2576 generic.go:356] "Generic (PLEG): container finished" podID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" containerID="3a4771437ce35e7d9ae5c20dad06223979cc9377c71539e0856626a68a0751f1" exitCode=0 Jul 24 23:02:04.334213 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:04.333835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825557984720" event={"ID":"4cb3462b-416c-44b4-9d4c-cc718e93dc02","Type":"ContainerDied","Data":"3a4771437ce35e7d9ae5c20dad06223979cc9377c71539e0856626a68a0751f1"} Jul 24 23:02:05.480754 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.480726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.500756 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.500699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825557984720_4cb3462b-416c-44b4-9d4c-cc718e93dc02/test-curl-7537-139825557984720/0.log" Jul 24 23:02:05.529905 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.529870 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:05.533636 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.533610 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:05.615953 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.615863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmjpk\" (UniqueName: \"kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk\") pod \"4cb3462b-416c-44b4-9d4c-cc718e93dc02\" (UID: \"4cb3462b-416c-44b4-9d4c-cc718e93dc02\") " Jul 24 23:02:05.618081 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.618050 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk" (OuterVolumeSpecName: "kube-api-access-lmjpk") pod "4cb3462b-416c-44b4-9d4c-cc718e93dc02" (UID: "4cb3462b-416c-44b4-9d4c-cc718e93dc02"). InnerVolumeSpecName "kube-api-access-lmjpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:02:05.717597 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:05.717557 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjpk\" (UniqueName: \"kubernetes.io/projected/4cb3462b-416c-44b4-9d4c-cc718e93dc02-kube-api-access-lmjpk\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:02:06.346673 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:06.346643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:06.346853 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:06.346680 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70ff23d181a25960091c2c948f399c92ebee47021ee65855ad305c99260dc37" Jul 24 23:02:06.356421 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:02:06.356387 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-139825557984720\" is forbidden: User \"system:node:ip-10-0-139-136.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-136.ec2.internal' and this object" podUID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:07.182039 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:07.182005 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" path="/var/lib/kubelet/pods/4cb3462b-416c-44b4-9d4c-cc718e93dc02/volumes" Jul 24 23:02:08.641548 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:08.641514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:08.938058 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:08.937963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:09.279744 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:09.279707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:11.909763 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:11.909717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:12.011218 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.011182 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:02:12.011756 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.011736 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" containerName="test-curl-7537-139825557984720" Jul 24 23:02:12.011756 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.011758 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" containerName="test-curl-7537-139825557984720" Jul 24 23:02:12.011917 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.011856 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb3462b-416c-44b4-9d4c-cc718e93dc02" containerName="test-curl-7537-139825557984720" Jul 24 23:02:12.012743 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.012715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:12.017262 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.017243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.020817 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.020795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e3f0a9-serving-cert\"" Jul 24 23:02:12.028548 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.028526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:02:12.078549 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.078514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:12.181003 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.180905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.181173 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.181044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.181173 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.181078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhs4\" (UniqueName: \"kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.282127 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.282085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.282369 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.282234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.282369 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.282271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhs4\" (UniqueName: \"kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.282853 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.282825 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.284795 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.284771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.290725 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.290699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhs4\" (UniqueName: \"kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4\") pod \"maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.305010 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.304981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:12.328707 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.328669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:12.475623 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.475595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:02:12.477911 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:02:12.477887 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab045d75_c2b0_4fbf_9b5b_e8665a83eee9.slice/crio-74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4 WatchSource:0}: Error finding container 74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4: Status 404 returned error can't find the container with id 74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4 Jul 24 23:02:12.568492 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.568459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:12.666825 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:12.666788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:13.396910 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:13.396868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" event={"ID":"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9","Type":"ContainerStarted","Data":"74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4"} Jul 24 23:02:14.404797 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:14.404744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" event={"ID":"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9","Type":"ContainerStarted","Data":"d0a20e5f596d565ecd6dba8196a24f3c7571d05813bbcd5ad7165ee1b09ec57c"} Jul 24 23:02:14.405279 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:14.404855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:14.422817 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:14.422764 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" podStartSLOduration=2.168025844 podStartE2EDuration="3.422748494s" totalImagesPullingTime="1.25472265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:02:14.42086009 +0000 UTC m=+2841.783299412" watchObservedRunningTime="2026-07-24 23:02:14.422748494 +0000 UTC m=+2841.785187815" Jul 24 23:02:16.371048 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:16.370993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:20.416074 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:20.416047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:02:27.253754 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:27.253720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:27.555068 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:27.554971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:28.122431 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:28.122391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.020451 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.020413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.099153 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.099121 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:02:31.101343 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.101283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.103223 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.103200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.105716 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.105695 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c6e95b-serving-cert\"" Jul 24 23:02:31.113861 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.113838 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:02:31.163453 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.163416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6827\" (UniqueName: \"kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.163453 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.163461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.163678 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.163592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.166177 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.166145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.264659 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.264622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6827\" (UniqueName: \"kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.264868 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.264667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.264868 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.264734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.264999 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:02:31.264862 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-serving-cert: secret "maas-api-e2e-ipp-b-c6e95b-serving-cert" not found Jul 24 23:02:31.264999 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:02:31.264940 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls podName:cdd21205-4f69-49af-8b41-cf836619ab70 nodeName:}" failed. No retries permitted until 2026-07-24 23:02:31.764918577 +0000 UTC m=+2859.127357893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls") pod "maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" (UID: "cdd21205-4f69-49af-8b41-cf836619ab70") : secret "maas-api-e2e-ipp-b-c6e95b-serving-cert" not found Jul 24 23:02:31.265505 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.265484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.273460 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.273395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6827\" (UniqueName: \"kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.319040 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.319000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.419542 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.419502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.521814 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.521777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.769733 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.769703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.772447 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.772419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:31.841783 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.841747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:31.998778 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:31.998742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:32.017384 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:32.017342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:32.180140 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:32.180115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:02:32.181851 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:02:32.181820 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd21205_4f69_49af_8b41_cf836619ab70.slice/crio-cfa09dc01a50b0d7d22d3b841409a827d9c6af5b41e1c106acc9e730498981c1 WatchSource:0}: Error finding container cfa09dc01a50b0d7d22d3b841409a827d9c6af5b41e1c106acc9e730498981c1: Status 404 returned error can't find the container with id cfa09dc01a50b0d7d22d3b841409a827d9c6af5b41e1c106acc9e730498981c1 Jul 24 23:02:32.508149 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:32.508119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" event={"ID":"cdd21205-4f69-49af-8b41-cf836619ab70","Type":"ContainerStarted","Data":"cfa09dc01a50b0d7d22d3b841409a827d9c6af5b41e1c106acc9e730498981c1"} Jul 24 23:02:33.515995 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:33.515941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" event={"ID":"cdd21205-4f69-49af-8b41-cf836619ab70","Type":"ContainerStarted","Data":"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f"} Jul 24 23:02:33.516472 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:33.516014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:02:33.532572 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:33.532518 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" podStartSLOduration=2.1603223 podStartE2EDuration="2.532500781s" totalImagesPullingTime="372.178481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:02:33.531550028 +0000 UTC m=+2860.893989349" watchObservedRunningTime="2026-07-24 23:02:33.532500781 +0000 UTC m=+2860.894940101" Jul 24 23:02:35.520978 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:35.520938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:02:39.528108 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:02:39.528078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:03:03.749566 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:03.749533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:34.711718 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:34.711660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:35.866770 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:35.866731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:35.970201 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:35.970162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:38.329931 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.329889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:38.412731 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.412691 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:03:38.416693 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.416668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.418717 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.418686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:38.419168 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.419147 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-34f01f-serving-cert\"" Jul 24 23:03:38.425448 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.424994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:03:38.487616 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.487579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:38.517813 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.517783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.518004 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.517891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpkq\" (UniqueName: \"kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.518066 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.518014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.618529 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.618442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.618529 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.618498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.618780 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.618538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpkq\" (UniqueName: \"kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.618780 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:03:38.618660 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-serving-cert: secret "maas-api-e2e-ipp-cleanup-34f01f-serving-cert" not found Jul 24 23:03:38.618780 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:03:38.618743 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls podName:53b828df-4956-44fe-958b-9d5b6bf58c3d nodeName:}" failed. No retries permitted until 2026-07-24 23:03:39.11871968 +0000 UTC m=+2926.481158981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" (UID: "53b828df-4956-44fe-958b-9d5b6bf58c3d") : secret "maas-api-e2e-ipp-cleanup-34f01f-serving-cert" not found Jul 24 23:03:38.619203 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.619184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.627183 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.627162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpkq\" (UniqueName: \"kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:38.970697 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:38.970661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:39.058912 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.058873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:39.123452 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.123415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:39.126004 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.125975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:39.328885 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.328791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:39.478239 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.478143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:03:39.478725 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:03:39.478434 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b828df_4956_44fe_958b_9d5b6bf58c3d.slice/crio-ea0d4f2d977ff547197b7b5a1eaa6add0852f924adeab21cc7043e2df691af98 WatchSource:0}: Error finding container ea0d4f2d977ff547197b7b5a1eaa6add0852f924adeab21cc7043e2df691af98: Status 404 returned error can't find the container with id ea0d4f2d977ff547197b7b5a1eaa6add0852f924adeab21cc7043e2df691af98 Jul 24 23:03:39.904703 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:39.904667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" event={"ID":"53b828df-4956-44fe-958b-9d5b6bf58c3d","Type":"ContainerStarted","Data":"ea0d4f2d977ff547197b7b5a1eaa6add0852f924adeab21cc7043e2df691af98"} Jul 24 23:03:40.912483 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:40.912445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" event={"ID":"53b828df-4956-44fe-958b-9d5b6bf58c3d","Type":"ContainerStarted","Data":"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f"} Jul 24 23:03:40.912923 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:40.912591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:40.932255 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:40.932200 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" podStartSLOduration=2.480736739 podStartE2EDuration="2.932182248s" totalImagesPullingTime="451.445509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:03:40.929610231 +0000 UTC m=+2928.292049546" watchObservedRunningTime="2026-07-24 23:03:40.932182248 +0000 UTC m=+2928.294621568" Jul 24 23:03:42.819232 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:42.819194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:03:46.924731 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:46.924705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:03:53.704249 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.704203 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj"] Jul 24 23:03:53.708390 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.708365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:53.711615 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.711587 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:03:53.714528 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.714501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj"] Jul 24 23:03:53.767921 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.767887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj\" (UID: \"4e029786-2db0-4df0-b457-9025aaec950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:53.868723 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.868684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj\" (UID: \"4e029786-2db0-4df0-b457-9025aaec950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:53.869347 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:53.869329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj\" (UID: \"4e029786-2db0-4df0-b457-9025aaec950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:54.022750 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:54.022659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:54.162683 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:54.162658 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj"] Jul 24 23:03:54.164828 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:03:54.164798 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e029786_2db0_4df0_b457_9025aaec950a.slice/crio-613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f WatchSource:0}: Error finding container 613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f: Status 404 returned error can't find the container with id 613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f Jul 24 23:03:54.993239 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:54.993205 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e029786-2db0-4df0-b457-9025aaec950a" containerID="ee82bda3168407f71931b93a8a909d87be7b2ff747d44c839bfa1e72ef7b1fb7" exitCode=0 Jul 24 23:03:54.993645 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:54.993313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" event={"ID":"4e029786-2db0-4df0-b457-9025aaec950a","Type":"ContainerDied","Data":"ee82bda3168407f71931b93a8a909d87be7b2ff747d44c839bfa1e72ef7b1fb7"} Jul 24 23:03:54.993645 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:54.993366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" event={"ID":"4e029786-2db0-4df0-b457-9025aaec950a","Type":"ContainerStarted","Data":"613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f"} Jul 24 23:03:56.139564 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:56.139538 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:56.188999 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:56.188961 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca\") pod \"4e029786-2db0-4df0-b457-9025aaec950a\" (UID: \"4e029786-2db0-4df0-b457-9025aaec950a\") " Jul 24 23:03:56.189359 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:56.189339 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e029786-2db0-4df0-b457-9025aaec950a" (UID: "4e029786-2db0-4df0-b457-9025aaec950a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:03:56.290394 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:56.290279 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e029786-2db0-4df0-b457-9025aaec950a-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:03:57.016724 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:57.016692 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" Jul 24 23:03:57.016905 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:57.016737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj" event={"ID":"4e029786-2db0-4df0-b457-9025aaec950a","Type":"ContainerDied","Data":"613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f"} Jul 24 23:03:57.016905 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:03:57.016813 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613cf8787bea812a38d64c90b44a80c7ccd3cbadf77591c98724b395f1762f2f" Jul 24 23:04:04.169014 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.168981 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:04:04.169506 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.169264 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" podUID="53b828df-4956-44fe-958b-9d5b6bf58c3d" containerName="maas-api" containerID="cri-o://b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f" gracePeriod=30 Jul 24 23:04:04.198394 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.198354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:04.299090 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.299022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:04.437576 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.437549 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:04:04.569333 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.569264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca\") pod \"53b828df-4956-44fe-958b-9d5b6bf58c3d\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " Jul 24 23:04:04.569509 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.569399 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdpkq\" (UniqueName: \"kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq\") pod \"53b828df-4956-44fe-958b-9d5b6bf58c3d\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " Jul 24 23:04:04.569509 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.569481 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") pod \"53b828df-4956-44fe-958b-9d5b6bf58c3d\" (UID: \"53b828df-4956-44fe-958b-9d5b6bf58c3d\") " Jul 24 23:04:04.569732 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.569704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53b828df-4956-44fe-958b-9d5b6bf58c3d" (UID: "53b828df-4956-44fe-958b-9d5b6bf58c3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:04:04.569821 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.569799 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b828df-4956-44fe-958b-9d5b6bf58c3d-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:04.571632 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.571611 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq" (OuterVolumeSpecName: "kube-api-access-xdpkq") pod "53b828df-4956-44fe-958b-9d5b6bf58c3d" (UID: "53b828df-4956-44fe-958b-9d5b6bf58c3d"). InnerVolumeSpecName "kube-api-access-xdpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:04.571724 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.571658 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53b828df-4956-44fe-958b-9d5b6bf58c3d" (UID: "53b828df-4956-44fe-958b-9d5b6bf58c3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:04:04.670597 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.670510 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpkq\" (UniqueName: \"kubernetes.io/projected/53b828df-4956-44fe-958b-9d5b6bf58c3d-kube-api-access-xdpkq\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:04.670597 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:04.670539 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b828df-4956-44fe-958b-9d5b6bf58c3d-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:05.062841 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.062808 2576 generic.go:356] "Generic (PLEG): container finished" podID="53b828df-4956-44fe-958b-9d5b6bf58c3d" containerID="b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f" exitCode=0 Jul 24 23:04:05.063054 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.062858 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" Jul 24 23:04:05.063054 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.062866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" event={"ID":"53b828df-4956-44fe-958b-9d5b6bf58c3d","Type":"ContainerDied","Data":"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f"} Jul 24 23:04:05.063054 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.062907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn" event={"ID":"53b828df-4956-44fe-958b-9d5b6bf58c3d","Type":"ContainerDied","Data":"ea0d4f2d977ff547197b7b5a1eaa6add0852f924adeab21cc7043e2df691af98"} Jul 24 23:04:05.063054 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.062927 2576 scope.go:117] "RemoveContainer" containerID="b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f" Jul 24 23:04:05.072577 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.072558 2576 scope.go:117] "RemoveContainer" containerID="b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f" Jul 24 23:04:05.072838 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:04:05.072818 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f\": container with ID starting with b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f not found: ID does not exist" containerID="b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f" Jul 24 23:04:05.072900 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.072850 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f"} err="failed to get container status \"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f\": rpc error: code = NotFound desc = could not find container \"b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f\": container with ID starting with b5bb949684ea41c1ea537dce58f41203e277f4152932b621cdf1164d19c9931f not found: ID does not exist" Jul 24 23:04:05.085125 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.085090 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:04:05.090429 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.090407 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-34f01f-cb989564d-6cbhn"] Jul 24 23:04:05.181930 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:05.181904 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b828df-4956-44fe-958b-9d5b6bf58c3d" path="/var/lib/kubelet/pods/53b828df-4956-44fe-958b-9d5b6bf58c3d/volumes" Jul 24 23:04:08.774493 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:08.774456 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj"] Jul 24 23:04:08.775745 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:08.775714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:08.776888 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:08.776868 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-34f01f-dbwxj"] Jul 24 23:04:09.146818 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:09.146706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:09.182312 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:09.182276 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e029786-2db0-4df0-b457-9025aaec950a" path="/var/lib/kubelet/pods/4e029786-2db0-4df0-b457-9025aaec950a/volumes" Jul 24 23:04:14.803226 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:14.803138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:14.902402 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:14.902356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:15.066535 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:15.066439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:15.285121 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:15.285079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:18.635705 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.635667 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n"] Jul 24 23:04:18.636133 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636112 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b828df-4956-44fe-958b-9d5b6bf58c3d" containerName="maas-api" Jul 24 23:04:18.636133 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636124 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b828df-4956-44fe-958b-9d5b6bf58c3d" containerName="maas-api" Jul 24 23:04:18.636216 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636140 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e029786-2db0-4df0-b457-9025aaec950a" containerName="revoke-keys" Jul 24 23:04:18.636216 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636146 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e029786-2db0-4df0-b457-9025aaec950a" containerName="revoke-keys" Jul 24 23:04:18.636283 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636231 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e029786-2db0-4df0-b457-9025aaec950a" containerName="revoke-keys" Jul 24 23:04:18.636283 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.636241 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b828df-4956-44fe-958b-9d5b6bf58c3d" containerName="maas-api" Jul 24 23:04:18.640836 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.640810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:18.643574 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.643544 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:04:18.646269 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.646209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n"] Jul 24 23:04:18.811788 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.811749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n\" (UID: \"eb59a89a-e135-42d5-8b42-c3ce3c82b716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:18.912880 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.912787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n\" (UID: \"eb59a89a-e135-42d5-8b42-c3ce3c82b716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:18.913447 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.913427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n\" (UID: \"eb59a89a-e135-42d5-8b42-c3ce3c82b716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:18.954367 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:18.954331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:19.100953 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:19.100567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n"] Jul 24 23:04:19.103246 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:04:19.103216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb59a89a_e135_42d5_8b42_c3ce3c82b716.slice/crio-e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6 WatchSource:0}: Error finding container e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6: Status 404 returned error can't find the container with id e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6 Jul 24 23:04:19.143600 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:19.143571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" event={"ID":"eb59a89a-e135-42d5-8b42-c3ce3c82b716","Type":"ContainerStarted","Data":"e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6"} Jul 24 23:04:20.150607 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:20.150510 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb59a89a-e135-42d5-8b42-c3ce3c82b716" containerID="d97e18017277272a7b54ebd71532c8549fafb9e8bb89c9cb74f819a42f451fb6" exitCode=0 Jul 24 23:04:20.150607 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:20.150579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" event={"ID":"eb59a89a-e135-42d5-8b42-c3ce3c82b716","Type":"ContainerDied","Data":"d97e18017277272a7b54ebd71532c8549fafb9e8bb89c9cb74f819a42f451fb6"} Jul 24 23:04:21.304018 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:21.303989 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:21.336070 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:21.336042 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca\") pod \"eb59a89a-e135-42d5-8b42-c3ce3c82b716\" (UID: \"eb59a89a-e135-42d5-8b42-c3ce3c82b716\") " Jul 24 23:04:21.336425 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:21.336403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb59a89a-e135-42d5-8b42-c3ce3c82b716" (UID: "eb59a89a-e135-42d5-8b42-c3ce3c82b716"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:04:21.437017 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:21.436930 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb59a89a-e135-42d5-8b42-c3ce3c82b716-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:22.162865 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:22.162826 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" Jul 24 23:04:22.163051 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:22.162835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n" event={"ID":"eb59a89a-e135-42d5-8b42-c3ce3c82b716","Type":"ContainerDied","Data":"e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6"} Jul 24 23:04:22.163051 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:22.162984 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e76f2c52aef5c1d6ee081ab96af8f67e35acd0c58e8b8f291b2df8b58322a6f6" Jul 24 23:04:33.984047 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:33.984009 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:04:33.984507 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:33.984265 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" podUID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" containerName="maas-api" containerID="cri-o://d0a20e5f596d565ecd6dba8196a24f3c7571d05813bbcd5ad7165ee1b09ec57c" gracePeriod=30 Jul 24 23:04:34.009453 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.009418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:34.073242 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.073195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:34.135349 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.135311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:34.235510 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.235427 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" containerID="d0a20e5f596d565ecd6dba8196a24f3c7571d05813bbcd5ad7165ee1b09ec57c" exitCode=0 Jul 24 23:04:34.235635 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.235506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" event={"ID":"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9","Type":"ContainerDied","Data":"d0a20e5f596d565ecd6dba8196a24f3c7571d05813bbcd5ad7165ee1b09ec57c"} Jul 24 23:04:34.235635 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.235564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" event={"ID":"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9","Type":"ContainerDied","Data":"74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4"} Jul 24 23:04:34.235635 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.235582 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d9fb6af36611c22ff354e49ba3d688f70c04dd8ec2b04769708ebf913fd3a4" Jul 24 23:04:34.240527 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.240507 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:04:34.359656 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.359630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfhs4\" (UniqueName: \"kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4\") pod \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " Jul 24 23:04:34.359828 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.359691 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls\") pod \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " Jul 24 23:04:34.359828 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.359758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca\") pod \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\" (UID: \"ab045d75-c2b0-4fbf-9b5b-e8665a83eee9\") " Jul 24 23:04:34.360217 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.360185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" (UID: "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:04:34.361946 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.361924 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4" (OuterVolumeSpecName: "kube-api-access-bfhs4") pod "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" (UID: "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9"). InnerVolumeSpecName "kube-api-access-bfhs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:34.362036 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.361980 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" (UID: "ab045d75-c2b0-4fbf-9b5b-e8665a83eee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:04:34.460616 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.460579 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhs4\" (UniqueName: \"kubernetes.io/projected/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-kube-api-access-bfhs4\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:34.460616 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.460608 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:34.460616 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:34.460618 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:04:35.241118 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:35.241083 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr" Jul 24 23:04:35.259864 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:35.259836 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:04:35.263590 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:35.263561 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3f0a9-5b48895d7-sjfwr"] Jul 24 23:04:37.183182 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:37.183148 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" path="/var/lib/kubelet/pods/ab045d75-c2b0-4fbf-9b5b-e8665a83eee9/volumes" Jul 24 23:04:38.698859 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:38.698818 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n"] Jul 24 23:04:38.703533 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:38.703509 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3f0a9-zt45n"] Jul 24 23:04:38.707187 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:38.707159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:39.056897 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:39.056796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:39.182480 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:39.182448 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb59a89a-e135-42d5-8b42-c3ce3c82b716" path="/var/lib/kubelet/pods/eb59a89a-e135-42d5-8b42-c3ce3c82b716/volumes" Jul 24 23:04:44.144357 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:44.144323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:04:53.333649 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:53.333623 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 23:04:53.340966 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:53.340941 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 23:04:53.342621 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:53.342603 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 23:04:53.349666 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:04:53.349647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2ffrc_849c3974-1299-4e99-afd3-1c0691ad5d88/ovn-acl-logging/0.log" Jul 24 23:05:20.588159 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:20.588128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:21.350969 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:21.350940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:26.895453 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:26.895410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:26.950165 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:26.950122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:27.164185 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.164089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:27.393971 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.393928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:27.516418 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516387 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g"] Jul 24 23:05:27.516852 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516836 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb59a89a-e135-42d5-8b42-c3ce3c82b716" containerName="revoke-keys" Jul 24 23:05:27.516852 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516851 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb59a89a-e135-42d5-8b42-c3ce3c82b716" containerName="revoke-keys" Jul 24 23:05:27.517015 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516874 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" containerName="maas-api" Jul 24 23:05:27.517015 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516879 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" containerName="maas-api" Jul 24 23:05:27.517015 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516950 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab045d75-c2b0-4fbf-9b5b-e8665a83eee9" containerName="maas-api" Jul 24 23:05:27.517015 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.516959 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb59a89a-e135-42d5-8b42-c3ce3c82b716" containerName="revoke-keys" Jul 24 23:05:27.520240 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.520223 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:27.522739 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.522723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:05:27.527181 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.527157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g"] Jul 24 23:05:27.684663 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.684617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g\" (UID: \"39d4d818-00b2-461e-8b21-50c183e5b698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:27.785315 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.785197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g\" (UID: \"39d4d818-00b2-461e-8b21-50c183e5b698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:27.785882 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.785854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g\" (UID: \"39d4d818-00b2-461e-8b21-50c183e5b698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:27.834291 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.834253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:27.975745 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:27.975713 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g"] Jul 24 23:05:27.978066 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:05:27.978040 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d4d818_00b2_461e_8b21_50c183e5b698.slice/crio-eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d WatchSource:0}: Error finding container eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d: Status 404 returned error can't find the container with id eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d Jul 24 23:05:28.556335 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:28.556214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" event={"ID":"39d4d818-00b2-461e-8b21-50c183e5b698","Type":"ContainerStarted","Data":"d575ad35deb632dd43c33660957bdaf45f48c84a7bbc824cba275a3fde3005b9"} Jul 24 23:05:28.556335 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:28.556273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" event={"ID":"39d4d818-00b2-461e-8b21-50c183e5b698","Type":"ContainerStarted","Data":"eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d"} Jul 24 23:05:28.573167 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:28.573117 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" podStartSLOduration=1.5731014509999999 podStartE2EDuration="1.573101451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:05:28.57132775 +0000 UTC m=+3035.933767070" watchObservedRunningTime="2026-07-24 23:05:28.573101451 +0000 UTC m=+3035.935540771" Jul 24 23:05:29.563592 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:29.563562 2576 generic.go:356] "Generic (PLEG): container finished" podID="39d4d818-00b2-461e-8b21-50c183e5b698" containerID="d575ad35deb632dd43c33660957bdaf45f48c84a7bbc824cba275a3fde3005b9" exitCode=0 Jul 24 23:05:29.564006 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:29.563624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" event={"ID":"39d4d818-00b2-461e-8b21-50c183e5b698","Type":"ContainerDied","Data":"d575ad35deb632dd43c33660957bdaf45f48c84a7bbc824cba275a3fde3005b9"} Jul 24 23:05:30.713440 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:30.713410 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:30.819478 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:30.819445 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca\") pod \"39d4d818-00b2-461e-8b21-50c183e5b698\" (UID: \"39d4d818-00b2-461e-8b21-50c183e5b698\") " Jul 24 23:05:30.819831 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:30.819782 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39d4d818-00b2-461e-8b21-50c183e5b698" (UID: "39d4d818-00b2-461e-8b21-50c183e5b698"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:30.920696 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:30.920662 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d4d818-00b2-461e-8b21-50c183e5b698-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:05:31.576273 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:31.576238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" event={"ID":"39d4d818-00b2-461e-8b21-50c183e5b698","Type":"ContainerDied","Data":"eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d"} Jul 24 23:05:31.576273 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:31.576266 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g" Jul 24 23:05:31.576501 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:31.576311 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb61090ab911a7eb2e9a9c9070674277792ac576f46b0d891900dcb7b82e7a9d" Jul 24 23:05:38.087344 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.087286 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:05:38.087794 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.087566 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" podUID="cdd21205-4f69-49af-8b41-cf836619ab70" containerName="maas-api" containerID="cri-o://abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f" gracePeriod=30 Jul 24 23:05:38.120661 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.120626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:38.219103 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.219066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:38.368212 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.368187 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:05:38.400761 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.400733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca\") pod \"cdd21205-4f69-49af-8b41-cf836619ab70\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " Jul 24 23:05:38.400941 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.400837 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") pod \"cdd21205-4f69-49af-8b41-cf836619ab70\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " Jul 24 23:05:38.400941 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.400932 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6827\" (UniqueName: \"kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827\") pod \"cdd21205-4f69-49af-8b41-cf836619ab70\" (UID: \"cdd21205-4f69-49af-8b41-cf836619ab70\") " Jul 24 23:05:38.401175 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.401152 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cdd21205-4f69-49af-8b41-cf836619ab70" (UID: "cdd21205-4f69-49af-8b41-cf836619ab70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:38.401243 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.401221 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21205-4f69-49af-8b41-cf836619ab70-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:05:38.403115 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.403081 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827" (OuterVolumeSpecName: "kube-api-access-w6827") pod "cdd21205-4f69-49af-8b41-cf836619ab70" (UID: "cdd21205-4f69-49af-8b41-cf836619ab70"). InnerVolumeSpecName "kube-api-access-w6827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:38.403271 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.403125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cdd21205-4f69-49af-8b41-cf836619ab70" (UID: "cdd21205-4f69-49af-8b41-cf836619ab70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:05:38.502613 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.502586 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd21205-4f69-49af-8b41-cf836619ab70-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:05:38.502613 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.502612 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6827\" (UniqueName: \"kubernetes.io/projected/cdd21205-4f69-49af-8b41-cf836619ab70-kube-api-access-w6827\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:05:38.615037 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.614939 2576 generic.go:356] "Generic (PLEG): container finished" podID="cdd21205-4f69-49af-8b41-cf836619ab70" containerID="abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f" exitCode=0 Jul 24 23:05:38.615037 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.614993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" Jul 24 23:05:38.615226 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.615030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" event={"ID":"cdd21205-4f69-49af-8b41-cf836619ab70","Type":"ContainerDied","Data":"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f"} Jul 24 23:05:38.615226 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.615089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8" event={"ID":"cdd21205-4f69-49af-8b41-cf836619ab70","Type":"ContainerDied","Data":"cfa09dc01a50b0d7d22d3b841409a827d9c6af5b41e1c106acc9e730498981c1"} Jul 24 23:05:38.615226 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.615111 2576 scope.go:117] "RemoveContainer" containerID="abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f" Jul 24 23:05:38.624807 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.624786 2576 scope.go:117] "RemoveContainer" containerID="abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f" Jul 24 23:05:38.625073 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:05:38.625056 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f\": container with ID starting with abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f not found: ID does not exist" containerID="abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f" Jul 24 23:05:38.625121 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.625085 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f"} err="failed to get container status \"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f\": rpc error: code = NotFound desc = could not find container \"abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f\": container with ID starting with abf43a7a7f7ba2229f22086fb2f92802ee94cff54e3921c47cd590cc8f39d59f not found: ID does not exist" Jul 24 23:05:38.637323 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.637271 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:05:38.640790 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:38.640768 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6e95b-56f55c8984-zxsr8"] Jul 24 23:05:39.182670 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:39.182640 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd21205-4f69-49af-8b41-cf836619ab70" path="/var/lib/kubelet/pods/cdd21205-4f69-49af-8b41-cf836619ab70/volumes" Jul 24 23:05:42.575043 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:42.575000 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g"] Jul 24 23:05:42.577256 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:42.577230 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c6e95b-qp84g"] Jul 24 23:05:42.581789 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:42.581765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:42.941669 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:42.941634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:43.181830 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:43.181750 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d4d818-00b2-461e-8b21-50c183e5b698" path="/var/lib/kubelet/pods/39d4d818-00b2-461e-8b21-50c183e5b698/volumes" Jul 24 23:05:48.603103 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:48.603066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:48.702780 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:48.702742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:48.871874 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:48.871768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:49.087916 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.087878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:49.216710 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.216676 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6"] Jul 24 23:05:49.217105 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217093 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d4d818-00b2-461e-8b21-50c183e5b698" containerName="revoke-keys" Jul 24 23:05:49.217146 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217109 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d4d818-00b2-461e-8b21-50c183e5b698" containerName="revoke-keys" Jul 24 23:05:49.217146 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217121 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd21205-4f69-49af-8b41-cf836619ab70" containerName="maas-api" Jul 24 23:05:49.217146 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd21205-4f69-49af-8b41-cf836619ab70" containerName="maas-api" Jul 24 23:05:49.217243 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217202 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d4d818-00b2-461e-8b21-50c183e5b698" containerName="revoke-keys" Jul 24 23:05:49.217243 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.217215 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd21205-4f69-49af-8b41-cf836619ab70" containerName="maas-api" Jul 24 23:05:49.221656 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.221639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:49.224379 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.224288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:05:49.226331 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.226277 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6"] Jul 24 23:05:49.306911 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.306874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6\" (UID: \"e528140b-e376-497d-a416-fa08f92ae561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:49.408206 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.408171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6\" (UID: \"e528140b-e376-497d-a416-fa08f92ae561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:49.408769 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.408751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6\" (UID: \"e528140b-e376-497d-a416-fa08f92ae561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:49.534026 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.533935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:49.678781 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:49.678752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6"] Jul 24 23:05:49.680909 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:05:49.680886 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode528140b_e376_497d_a416_fa08f92ae561.slice/crio-0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d WatchSource:0}: Error finding container 0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d: Status 404 returned error can't find the container with id 0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d Jul 24 23:05:50.687148 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:50.687053 2576 generic.go:356] "Generic (PLEG): container finished" podID="e528140b-e376-497d-a416-fa08f92ae561" containerID="0620bf98e89f6391c41f3b661dc809841b841fae765abe9fd58c346988975d86" exitCode=0 Jul 24 23:05:50.687148 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:50.687121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" event={"ID":"e528140b-e376-497d-a416-fa08f92ae561","Type":"ContainerDied","Data":"0620bf98e89f6391c41f3b661dc809841b841fae765abe9fd58c346988975d86"} Jul 24 23:05:50.687618 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:50.687169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" event={"ID":"e528140b-e376-497d-a416-fa08f92ae561","Type":"ContainerStarted","Data":"0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d"} Jul 24 23:05:51.841408 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:51.841375 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:51.931482 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:51.931447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca\") pod \"e528140b-e376-497d-a416-fa08f92ae561\" (UID: \"e528140b-e376-497d-a416-fa08f92ae561\") " Jul 24 23:05:51.931858 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:51.931830 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e528140b-e376-497d-a416-fa08f92ae561" (UID: "e528140b-e376-497d-a416-fa08f92ae561"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:52.033054 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.032972 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e528140b-e376-497d-a416-fa08f92ae561-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:05:52.701233 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.701202 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" Jul 24 23:05:52.701233 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.701219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6" event={"ID":"e528140b-e376-497d-a416-fa08f92ae561","Type":"ContainerDied","Data":"0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d"} Jul 24 23:05:52.701468 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.701277 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aa8f0dd69122100554e49a67dc2e82241933d69592b3d1160225b0c31ef127d" Jul 24 23:05:52.817819 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.817786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:52.878669 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:52.878631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:59.545202 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.545150 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 23:05:59.545750 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.545424 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" podUID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" containerName="maas-api" containerID="cri-o://a8117ff69156a905f55118baf25ef64e01b99d7fa8401afee0c16d01d49641cd" gracePeriod=30 Jul 24 23:05:59.570661 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.570605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:59.631204 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.631142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:59.731553 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.731518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:05:59.745068 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.745035 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" containerID="a8117ff69156a905f55118baf25ef64e01b99d7fa8401afee0c16d01d49641cd" exitCode=0 Jul 24 23:05:59.745242 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.745125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" event={"ID":"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1","Type":"ContainerDied","Data":"a8117ff69156a905f55118baf25ef64e01b99d7fa8401afee0c16d01d49641cd"} Jul 24 23:05:59.801834 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.801757 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 23:05:59.907529 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.907497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca\") pod \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " Jul 24 23:05:59.907760 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.907553 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4q4z\" (UniqueName: \"kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z\") pod \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " Jul 24 23:05:59.907760 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.907631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") pod \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\" (UID: \"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1\") " Jul 24 23:05:59.907920 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.907894 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" (UID: "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:59.909852 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.909824 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z" (OuterVolumeSpecName: "kube-api-access-v4q4z") pod "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" (UID: "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1"). InnerVolumeSpecName "kube-api-access-v4q4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:59.909964 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:05:59.909848 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" (UID: "5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:06:00.008920 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.008879 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:00.008920 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.008913 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:00.008920 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.008929 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4q4z\" (UniqueName: \"kubernetes.io/projected/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1-kube-api-access-v4q4z\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:00.751654 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.751618 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" Jul 24 23:06:00.751654 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.751650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5" event={"ID":"5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1","Type":"ContainerDied","Data":"3a1655d5ecb156ae32b78d40d192aab72180a1b0547f68adee84b83e2543690a"} Jul 24 23:06:00.752123 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.751720 2576 scope.go:117] "RemoveContainer" containerID="a8117ff69156a905f55118baf25ef64e01b99d7fa8401afee0c16d01d49641cd" Jul 24 23:06:00.773496 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.773468 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 23:06:00.776815 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:00.776790 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-499dd1-7878bc5fc8-wbpx5"] Jul 24 23:06:01.181971 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:01.181892 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" path="/var/lib/kubelet/pods/5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1/volumes" Jul 24 23:06:04.279927 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:04.279883 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6"] Jul 24 23:06:04.282878 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:04.282853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:04.283050 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:04.283032 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-499dd1-vfqt6"] Jul 24 23:06:04.653408 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:04.653241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:05.182460 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:05.182423 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e528140b-e376-497d-a416-fa08f92ae561" path="/var/lib/kubelet/pods/e528140b-e376-497d-a416-fa08f92ae561/volumes" Jul 24 23:06:10.454764 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:10.454731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:10.556250 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:10.556199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:10.723394 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:10.723265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:10.955004 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:10.954967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:11.074074 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.073987 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx"] Jul 24 23:06:11.074437 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074424 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" containerName="maas-api" Jul 24 23:06:11.074490 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074439 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" containerName="maas-api" Jul 24 23:06:11.074490 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074449 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e528140b-e376-497d-a416-fa08f92ae561" containerName="revoke-keys" Jul 24 23:06:11.074490 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074454 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e528140b-e376-497d-a416-fa08f92ae561" containerName="revoke-keys" Jul 24 23:06:11.074589 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074525 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf1bc44-8862-4e0f-a297-6ddcbcc36aa1" containerName="maas-api" Jul 24 23:06:11.074589 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.074532 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e528140b-e376-497d-a416-fa08f92ae561" containerName="revoke-keys" Jul 24 23:06:11.077835 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.077819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:11.080632 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.080590 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcbdx\"" Jul 24 23:06:11.083912 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.083889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx"] Jul 24 23:06:11.221674 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.221641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx\" (UID: \"b656d714-dc17-4531-9523-f9961f648741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:11.322726 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.322693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx\" (UID: \"b656d714-dc17-4531-9523-f9961f648741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:11.323277 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.323255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx\" (UID: \"b656d714-dc17-4531-9523-f9961f648741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:11.390331 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.390208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:11.532104 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.532082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx"] Jul 24 23:06:11.535013 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:06:11.534980 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb656d714_dc17_4531_9523_f9961f648741.slice/crio-3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97 WatchSource:0}: Error finding container 3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97: Status 404 returned error can't find the container with id 3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97 Jul 24 23:06:11.816899 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.816857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" event={"ID":"b656d714-dc17-4531-9523-f9961f648741","Type":"ContainerStarted","Data":"a5b9a02ff54081aa15764366294512ae4468f4a7dbf16bc4d6ab572665e202ba"} Jul 24 23:06:11.817079 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.816926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" event={"ID":"b656d714-dc17-4531-9523-f9961f648741","Type":"ContainerStarted","Data":"3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97"} Jul 24 23:06:11.831339 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:11.831258 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" podStartSLOduration=0.831241169 podStartE2EDuration="831.241169ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:06:11.830357965 +0000 UTC m=+3079.192797286" watchObservedRunningTime="2026-07-24 23:06:11.831241169 +0000 UTC m=+3079.193680512" Jul 24 23:06:12.825151 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:12.825109 2576 generic.go:356] "Generic (PLEG): container finished" podID="b656d714-dc17-4531-9523-f9961f648741" containerID="a5b9a02ff54081aa15764366294512ae4468f4a7dbf16bc4d6ab572665e202ba" exitCode=0 Jul 24 23:06:12.825598 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:12.825196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" event={"ID":"b656d714-dc17-4531-9523-f9961f648741","Type":"ContainerDied","Data":"a5b9a02ff54081aa15764366294512ae4468f4a7dbf16bc4d6ab572665e202ba"} Jul 24 23:06:13.974703 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:13.974678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:14.047065 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.046573 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca\") pod \"b656d714-dc17-4531-9523-f9961f648741\" (UID: \"b656d714-dc17-4531-9523-f9961f648741\") " Jul 24 23:06:14.047065 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.047004 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b656d714-dc17-4531-9523-f9961f648741" (UID: "b656d714-dc17-4531-9523-f9961f648741"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:06:14.047385 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.047244 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b656d714-dc17-4531-9523-f9961f648741-maas-api-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:14.838022 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.837992 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" Jul 24 23:06:14.838203 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.837995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx" event={"ID":"b656d714-dc17-4531-9523-f9961f648741","Type":"ContainerDied","Data":"3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97"} Jul 24 23:06:14.838203 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:14.838115 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bd6c44e92366d2f8f8854e7fa05d5d31c10cfbb07f613bd23a6de7e455b8f97" Jul 24 23:06:21.586928 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.586894 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 23:06:21.587429 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.587123 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" podUID="56da2490-0f84-4d74-8b05-f0d783d299be" containerName="maas-api" containerID="cri-o://53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75" gracePeriod=30 Jul 24 23:06:21.610570 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.610544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:21.673270 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.673234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:21.773262 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.773234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:21.845470 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.845405 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 23:06:21.879218 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.879187 2576 generic.go:356] "Generic (PLEG): container finished" podID="56da2490-0f84-4d74-8b05-f0d783d299be" containerID="53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75" exitCode=0 Jul 24 23:06:21.879423 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.879234 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" Jul 24 23:06:21.879423 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.879283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" event={"ID":"56da2490-0f84-4d74-8b05-f0d783d299be","Type":"ContainerDied","Data":"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75"} Jul 24 23:06:21.879423 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.879380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr" event={"ID":"56da2490-0f84-4d74-8b05-f0d783d299be","Type":"ContainerDied","Data":"d06dc3fe6040a7f23103a511c0d940c0b574329606843512a1cf8fabdb922633"} Jul 24 23:06:21.879423 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.879417 2576 scope.go:117] "RemoveContainer" containerID="53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75" Jul 24 23:06:21.889631 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.889612 2576 scope.go:117] "RemoveContainer" containerID="53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75" Jul 24 23:06:21.889917 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:06:21.889898 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75\": container with ID starting with 53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75 not found: ID does not exist" containerID="53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75" Jul 24 23:06:21.889978 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.889927 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75"} err="failed to get container status \"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75\": rpc error: code = NotFound desc = could not find container \"53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75\": container with ID starting with 53ef04dda8bdff138adf52844ccf6368491071604ac1f31282ad1b014c498d75 not found: ID does not exist" Jul 24 23:06:21.923485 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.923452 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") pod \"56da2490-0f84-4d74-8b05-f0d783d299be\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " Jul 24 23:06:21.923485 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.923489 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca\") pod \"56da2490-0f84-4d74-8b05-f0d783d299be\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " Jul 24 23:06:21.923723 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.923570 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5djbh\" (UniqueName: \"kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh\") pod \"56da2490-0f84-4d74-8b05-f0d783d299be\" (UID: \"56da2490-0f84-4d74-8b05-f0d783d299be\") " Jul 24 23:06:21.923917 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.923892 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56da2490-0f84-4d74-8b05-f0d783d299be" (UID: "56da2490-0f84-4d74-8b05-f0d783d299be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:06:21.925731 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.925711 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56da2490-0f84-4d74-8b05-f0d783d299be" (UID: "56da2490-0f84-4d74-8b05-f0d783d299be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:06:21.925872 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:21.925856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh" (OuterVolumeSpecName: "kube-api-access-5djbh") pod "56da2490-0f84-4d74-8b05-f0d783d299be" (UID: "56da2490-0f84-4d74-8b05-f0d783d299be"). InnerVolumeSpecName "kube-api-access-5djbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:06:22.024837 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:22.024805 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56da2490-0f84-4d74-8b05-f0d783d299be-maas-api-tls\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:22.024837 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:22.024834 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56da2490-0f84-4d74-8b05-f0d783d299be-openshift-service-ca\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:22.024837 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:22.024844 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djbh\" (UniqueName: \"kubernetes.io/projected/56da2490-0f84-4d74-8b05-f0d783d299be-kube-api-access-5djbh\") on node \"ip-10-0-139-136.ec2.internal\" DevicePath \"\"" Jul 24 23:06:22.208864 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:22.208824 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 23:06:22.212188 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:22.212161 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e15986-56f7775cf8-qh6sr"] Jul 24 23:06:22.231059 ip-10-0-139-136 kubenswrapper[2576]: E0724 23:06:22.231029 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56da2490_0f84_4d74_8b05_f0d783d299be.slice/crio-d06dc3fe6040a7f23103a511c0d940c0b574329606843512a1cf8fabdb922633\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56da2490_0f84_4d74_8b05_f0d783d299be.slice\": RecentStats: unable to find data in memory cache]" Jul 24 23:06:23.181930 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:23.181891 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56da2490-0f84-4d74-8b05-f0d783d299be" path="/var/lib/kubelet/pods/56da2490-0f84-4d74-8b05-f0d783d299be/volumes" Jul 24 23:06:26.136542 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:26.136496 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx"] Jul 24 23:06:26.138134 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:26.138106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:26.138890 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:26.138871 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e15986-lzvfx"] Jul 24 23:06:26.497122 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:26.497082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:27.181430 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:27.181393 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b656d714-dc17-4531-9523-f9961f648741" path="/var/lib/kubelet/pods/b656d714-dc17-4531-9523-f9961f648741/volumes" Jul 24 23:06:32.290960 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:32.290922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:32.365711 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:32.365668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:32.577224 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:32.577126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:32.767504 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:32.767468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:33.201072 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:33.201041 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6b445f89-kjqvv_c3953506-d593-4f75-8458-6832d4cfd8bc/authorino/0.log" Jul 24 23:06:38.333398 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:38.333363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9x4sp_d5a98dee-6290-4919-be2e-15fcaa8d5cc3/manager/0.log" Jul 24 23:06:38.697887 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:38.697861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mm5jq_2d3133ce-3898-4b25-8be9-e8df2c1c23d6/server/0.log" Jul 24 23:06:38.816284 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:38.816247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m82cl_ee29d1d3-691b-4191-aecc-54a0ae5499a7/manager/0.log" Jul 24 23:06:38.920822 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:38.920789 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-6px8j_460371d3-e56e-407b-b6e0-b03f31a2e380/copy-manifests/0.log" Jul 24 23:06:38.927571 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:38.927546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-6px8j_460371d3-e56e-407b-b6e0-b03f31a2e380/manager/0.log" Jul 24 23:06:39.510870 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:39.510827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bc7645cc-txtvl_78c7fee3-f730-42f3-ba04-8ac28c6b1ee7/maas-api/0.log" Jul 24 23:06:39.721710 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:39.721675 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748885-2946x_496cb305-14c2-43a3-9492-0dd2fde1c2e5/cleanup/0.log" Jul 24 23:06:39.822880 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:39.822803 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748900-7x7rr_3dedbfdf-067a-43f3-a56c-ed5285c9611a/cleanup/0.log" Jul 24 23:06:39.929568 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:39.929528 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vbbqx_e5ee7e87-0514-43ba-8008-252a9a8c7d15/postgres/0.log" Jul 24 23:06:41.160938 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.160908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6b445f89-kjqvv_c3953506-d593-4f75-8458-6832d4cfd8bc/authorino/0.log" Jul 24 23:06:41.267521 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.267493 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bvxmb_84081949-17e9-4332-a378-75706479ecd1/manager/0.log" Jul 24 23:06:41.386609 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.386577 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xlpk5_feb7fa2f-b93e-4f5e-822a-bb11f1c376da/manager/0.log" Jul 24 23:06:41.490819 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.490789 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rkfhs_348787a2-6446-4d92-a998-f3a000f2a712/kuadrant-console-plugin/0.log" Jul 24 23:06:41.611856 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.611822 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wmgc4_7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a/registry-server/0.log" Jul 24 23:06:41.733437 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.733402 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rmkfd_cbea030d-784e-422a-b694-200f53809491/manager/0.log" Jul 24 23:06:41.958367 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:41.958322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nf755_050e27c9-a99c-47b2-bc6f-5be844d06517/manager/0.log" Jul 24 23:06:42.632000 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:42.631964 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-q748r_7c3e254a-21fb-4fd9-b6d9-502086724a94/kube-auth-proxy/0.log" Jul 24 23:06:42.866777 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:42.866742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-49gcl_ad42bcf7-ee9a-440d-9abe-42a052d59018/payload-pre-processing/0.log" Jul 24 23:06:43.534889 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:43.534860 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qml8c_f9778b4e-e7f3-4f7a-a257-644ee9ee9897/main/0.log" Jul 24 23:06:43.750663 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:43.750629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn45wx_d1c7c1a6-d3e7-48cd-8b15-5bc565a954a8/main/0.log" Jul 24 23:06:49.124806 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.124771 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rsljv/must-gather-h527v"] Jul 24 23:06:49.125208 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125195 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b656d714-dc17-4531-9523-f9961f648741" containerName="revoke-keys" Jul 24 23:06:49.125208 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125209 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b656d714-dc17-4531-9523-f9961f648741" containerName="revoke-keys" Jul 24 23:06:49.125288 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125238 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56da2490-0f84-4d74-8b05-f0d783d299be" containerName="maas-api" Jul 24 23:06:49.125288 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125244 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="56da2490-0f84-4d74-8b05-f0d783d299be" containerName="maas-api" Jul 24 23:06:49.125371 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125327 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b656d714-dc17-4531-9523-f9961f648741" containerName="revoke-keys" Jul 24 23:06:49.125371 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.125340 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="56da2490-0f84-4d74-8b05-f0d783d299be" containerName="maas-api" Jul 24 23:06:49.128888 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.128858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.133035 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.133010 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsljv\"/\"kube-root-ca.crt\"" Jul 24 23:06:49.133539 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.133511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:49.134136 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.134119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsljv\"/\"openshift-service-ca.crt\"" Jul 24 23:06:49.134212 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.134162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rsljv\"/\"default-dockercfg-66cxv\"" Jul 24 23:06:49.140201 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.140177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/must-gather-h527v"] Jul 24 23:06:49.185371 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.185339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwvg5\" (UniqueName: \"kubernetes.io/projected/ff10e727-efbd-4454-83e9-5b93b559d98b-kube-api-access-rwvg5\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.185543 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.185397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ff10e727-efbd-4454-83e9-5b93b559d98b-must-gather-output\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.228751 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.228714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:06:49.285861 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.285822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwvg5\" (UniqueName: \"kubernetes.io/projected/ff10e727-efbd-4454-83e9-5b93b559d98b-kube-api-access-rwvg5\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.286032 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.285892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ff10e727-efbd-4454-83e9-5b93b559d98b-must-gather-output\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.286199 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.286184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ff10e727-efbd-4454-83e9-5b93b559d98b-must-gather-output\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.293769 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.293741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwvg5\" (UniqueName: \"kubernetes.io/projected/ff10e727-efbd-4454-83e9-5b93b559d98b-kube-api-access-rwvg5\") pod \"must-gather-h527v\" (UID: \"ff10e727-efbd-4454-83e9-5b93b559d98b\") " pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.439876 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.439785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/must-gather-h527v" Jul 24 23:06:49.577880 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:49.577854 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/must-gather-h527v"] Jul 24 23:06:49.579482 ip-10-0-139-136 kubenswrapper[2576]: W0724 23:06:49.579459 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff10e727_efbd_4454_83e9_5b93b559d98b.slice/crio-b9f6b6ff717a63bfb9eb94df6684fa608e33b2d6e58f9e39ff0a7f8f5c5d8778 WatchSource:0}: Error finding container b9f6b6ff717a63bfb9eb94df6684fa608e33b2d6e58f9e39ff0a7f8f5c5d8778: Status 404 returned error can't find the container with id b9f6b6ff717a63bfb9eb94df6684fa608e33b2d6e58f9e39ff0a7f8f5c5d8778 Jul 24 23:06:50.046141 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:50.046105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/must-gather-h527v" event={"ID":"ff10e727-efbd-4454-83e9-5b93b559d98b","Type":"ContainerStarted","Data":"b9f6b6ff717a63bfb9eb94df6684fa608e33b2d6e58f9e39ff0a7f8f5c5d8778"} Jul 24 23:06:51.058925 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:51.058883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/must-gather-h527v" event={"ID":"ff10e727-efbd-4454-83e9-5b93b559d98b","Type":"ContainerStarted","Data":"19bd8fa1835db3ead13a7dfc036addafecc9d81322c8e7979907aaae9d5d720f"} Jul 24 23:06:51.059413 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:51.058959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/must-gather-h527v" event={"ID":"ff10e727-efbd-4454-83e9-5b93b559d98b","Type":"ContainerStarted","Data":"97ecc25b9fba8e4d8614825cef3d808e069afa74aa8f8876351d30111fdf0429"} Jul 24 23:06:51.077139 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:51.077086 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rsljv/must-gather-h527v" podStartSLOduration=1.3338786329999999 podStartE2EDuration="2.077069365s" totalImagesPullingTime="743.190732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:06:51.07458465 +0000 UTC m=+3118.437023972" watchObservedRunningTime="2026-07-24 23:06:51.077069365 +0000 UTC m=+3118.439508693" Jul 24 23:06:51.816646 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:51.816617 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-84ckx_0980d09d-775a-4f2a-82d9-7a5167813f50/global-pull-secret-syncer/0.log" Jul 24 23:06:51.938413 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:51.938376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" Jul 24 23:06:52.034579 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:52.034549 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-136.ec2.internal_1b42fb50bb861bf003dffd0219a78cbf/haproxy/0.log" Jul 24 23:06:57.860876 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:57.860355 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6b445f89-kjqvv_c3953506-d593-4f75-8458-6832d4cfd8bc/authorino/0.log" Jul 24 23:06:57.890317 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:57.889929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bvxmb_84081949-17e9-4332-a378-75706479ecd1/manager/0.log" Jul 24 23:06:57.917328 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:57.916671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xlpk5_feb7fa2f-b93e-4f5e-822a-bb11f1c376da/manager/0.log" Jul 24 23:06:57.939803 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:57.939768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rkfhs_348787a2-6446-4d92-a998-f3a000f2a712/kuadrant-console-plugin/0.log" Jul 24 23:06:57.991347 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:57.991312 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wmgc4_7ba6767e-6a37-4499-aa42-5e4e7a5eaf4a/registry-server/0.log" Jul 24 23:06:58.269390 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:58.269354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rmkfd_cbea030d-784e-422a-b694-200f53809491/manager/0.log" Jul 24 23:06:58.421791 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:58.419842 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nf755_050e27c9-a99c-47b2-bc6f-5be844d06517/manager/0.log" Jul 24 23:06:59.970146 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:59.970119 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/alertmanager/0.log" Jul 24 23:06:59.987337 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:06:59.987311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/config-reloader/0.log" Jul 24 23:07:00.031269 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.031243 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/kube-rbac-proxy-web/0.log" Jul 24 23:07:00.072888 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.072797 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/kube-rbac-proxy/0.log" Jul 24 23:07:00.126846 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.126808 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/kube-rbac-proxy-metric/0.log" Jul 24 23:07:00.149833 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.149805 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/prom-label-proxy/0.log" Jul 24 23:07:00.168583 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.168558 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_421582e8-d17e-4ff7-9136-88e1aef78f35/init-config-reloader/0.log" Jul 24 23:07:00.225452 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.225368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mkmdn_7009454d-7544-4d3b-b842-921f3b7a55ea/kube-state-metrics/0.log" Jul 24 23:07:00.239781 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.239753 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mkmdn_7009454d-7544-4d3b-b842-921f3b7a55ea/kube-rbac-proxy-main/0.log" Jul 24 23:07:00.257365 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.257334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mkmdn_7009454d-7544-4d3b-b842-921f3b7a55ea/kube-rbac-proxy-self/0.log" Jul 24 23:07:00.327730 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.327656 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f22m_23330042-d32f-4094-afbf-64ff69e8ef93/node-exporter/0.log" Jul 24 23:07:00.344134 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.344105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f22m_23330042-d32f-4094-afbf-64ff69e8ef93/kube-rbac-proxy/0.log" Jul 24 23:07:00.360264 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.360239 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f22m_23330042-d32f-4094-afbf-64ff69e8ef93/init-textfile/0.log" Jul 24 23:07:00.680124 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.680035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-5bgvn_2810ffcd-958b-4210-b860-9239954d7df8/prometheus-operator/0.log" Jul 24 23:07:00.692629 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.692595 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-5bgvn_2810ffcd-958b-4210-b860-9239954d7df8/kube-rbac-proxy/0.log" Jul 24 23:07:00.739008 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.738981 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db4c9fc48-bjv99_42ed3bcb-99ad-4b62-ac6f-5a89af7361da/telemeter-client/0.log" Jul 24 23:07:00.765667 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.765535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db4c9fc48-bjv99_42ed3bcb-99ad-4b62-ac6f-5a89af7361da/reload/0.log" Jul 24 23:07:00.781710 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:00.781681 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db4c9fc48-bjv99_42ed3bcb-99ad-4b62-ac6f-5a89af7361da/kube-rbac-proxy/0.log" Jul 24 23:07:02.033749 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.033713 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr"] Jul 24 23:07:02.041550 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.041527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.046804 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.046775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr"] Jul 24 23:07:02.133395 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.133352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-podres\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.133603 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.133440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-proc\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.133603 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.133495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqld\" (UniqueName: \"kubernetes.io/projected/62437154-025a-4912-a043-db1529248999-kube-api-access-khqld\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.133603 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.133554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-sys\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.133800 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.133612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-lib-modules\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235179 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-sys\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235386 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-lib-modules\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235386 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-podres\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235386 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-sys\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235386 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-proc\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235561 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqld\" (UniqueName: \"kubernetes.io/projected/62437154-025a-4912-a043-db1529248999-kube-api-access-khqld\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235561 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-podres\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235561 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-lib-modules\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.235561 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.235505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62437154-025a-4912-a043-db1529248999-proc\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.249636 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.249602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqld\" (UniqueName: \"kubernetes.io/projected/62437154-025a-4912-a043-db1529248999-kube-api-access-khqld\") pod \"perf-node-gather-daemonset-rdggr\" (UID: \"62437154-025a-4912-a043-db1529248999\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.358087 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.357749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:02.586689 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.586176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr"] Jul 24 23:07:02.694657 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.694627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/2.log" Jul 24 23:07:02.702771 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:02.702724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-sdhsp_8a58ca8d-ae71-448c-8fd7-0a0f1ad98240/console-operator/3.log" Jul 24 23:07:03.185259 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:03.185221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" event={"ID":"62437154-025a-4912-a043-db1529248999","Type":"ContainerStarted","Data":"51c22867ca83fa9b28f515b431e49d657e3b89b03032fee44821c6abccdfbd20"} Jul 24 23:07:03.185259 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:03.185263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" event={"ID":"62437154-025a-4912-a043-db1529248999","Type":"ContainerStarted","Data":"1e218ed49414517600639fbcd41485df137113f67999eb8c4c48407f2a2eb9e3"} Jul 24 23:07:03.185879 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:03.185283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:03.189050 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:03.188972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c7d47cdf-w6ffk_5eded83e-9023-44b6-ae19-d7faa9678a0d/console/0.log" Jul 24 23:07:03.202544 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:03.202496 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" podStartSLOduration=1.202479357 podStartE2EDuration="1.202479357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:07:03.200251084 +0000 UTC m=+3130.562690408" watchObservedRunningTime="2026-07-24 23:07:03.202479357 +0000 UTC m=+3130.564918677" Jul 24 23:07:04.814036 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:04.814006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpljc_386afff3-1e58-4e1b-95be-2cc4087b78f7/dns/0.log" Jul 24 23:07:04.830241 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:04.830215 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpljc_386afff3-1e58-4e1b-95be-2cc4087b78f7/kube-rbac-proxy/0.log" Jul 24 23:07:04.891830 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:04.891799 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjcxr_6c3e650e-692a-4e51-910a-1772c02cb4bd/dns-node-resolver/0.log" Jul 24 23:07:05.402177 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:05.402146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7bf68ccf44-m4glj_a42c230f-289f-4e01-b929-9698544e66e7/registry/0.log" Jul 24 23:07:05.464378 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:05.464347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h9tgc_75481ec4-c026-4496-94fe-050f60955b2f/node-ca/0.log" Jul 24 23:07:06.512396 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:06.512371 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-q748r_7c3e254a-21fb-4fd9-b6d9-502086724a94/kube-auth-proxy/0.log" Jul 24 23:07:06.599148 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:06.599089 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-49gcl_ad42bcf7-ee9a-440d-9abe-42a052d59018/payload-pre-processing/0.log" Jul 24 23:07:07.481169 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:07.481125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lds7x_cf12a677-1a57-4dbf-803f-90f6bdd3b722/serve-healthcheck-canary/0.log" Jul 24 23:07:07.930344 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:07.930253 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d7hv_d4bb5c83-712e-430d-aea1-882e82e2294d/kube-rbac-proxy/0.log" Jul 24 23:07:07.946669 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:07.946641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d7hv_d4bb5c83-712e-430d-aea1-882e82e2294d/exporter/0.log" Jul 24 23:07:07.964019 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:07.963990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d7hv_d4bb5c83-712e-430d-aea1-882e82e2294d/extractor/0.log" Jul 24 23:07:09.205511 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:09.205479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-rdggr" Jul 24 23:07:10.072685 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:10.072572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9x4sp_d5a98dee-6290-4919-be2e-15fcaa8d5cc3/manager/0.log" Jul 24 23:07:10.409645 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:10.409561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mm5jq_2d3133ce-3898-4b25-8be9-e8df2c1c23d6/server/0.log" Jul 24 23:07:10.530399 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:10.530364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m82cl_ee29d1d3-691b-4191-aecc-54a0ae5499a7/manager/0.log" Jul 24 23:07:10.555147 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:10.555113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-6px8j_460371d3-e56e-407b-b6e0-b03f31a2e380/manager/0.log" Jul 24 23:07:10.573777 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:10.573739 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-6px8j_460371d3-e56e-407b-b6e0-b03f31a2e380/copy-manifests/0.log" Jul 24 23:07:12.410830 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:12.410738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6n9w6_3f7e2ba4-85bc-48de-af19-38e7bcababb4/konnectivity-agent/0.log" limit=1024 Jul 24 23:07:12.954875 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:12.954830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-q748r_7c3e254a-21fb-4fd9-b6d9-502086724a94/kube-auth-proxy/0.log" Jul 24 23:07:13.029638 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:13.029606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-49gcl_ad42bcf7-ee9a-440d-9abe-42a052d59018/payload-pre-processing/0.log" Jul 24 23:07:14.613227 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:14.613191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-q748r_7c3e254a-21fb-4fd9-b6d9-502086724a94/kube-auth-proxy/0.log" Jul 24 23:07:14.697742 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:14.697705 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-49gcl_ad42bcf7-ee9a-440d-9abe-42a052d59018/payload-pre-processing/0.log" Jul 24 23:07:17.437480 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:17.437443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jt654_33138bc3-3754-4211-85fc-ed955daaaa5f/kube-storage-version-migrator-operator/1.log" Jul 24 23:07:17.439238 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:17.439211 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jt654_33138bc3-3754-4211-85fc-ed955daaaa5f/kube-storage-version-migrator-operator/0.log" Jul 24 23:07:18.450429 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.450395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/kube-multus-additional-cni-plugins/0.log" Jul 24 23:07:18.469469 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.469443 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/egress-router-binary-copy/0.log" Jul 24 23:07:18.485883 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.485859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/cni-plugins/0.log" Jul 24 23:07:18.501138 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.501114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/bond-cni-plugin/0.log" Jul 24 23:07:18.517014 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.516986 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/routeoverride-cni/0.log" Jul 24 23:07:18.532259 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.532233 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/whereabouts-cni-bincopy/0.log" Jul 24 23:07:18.547381 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.547354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rkhl_afc8daf1-aced-453d-99be-31e6ca0a148d/whereabouts-cni/0.log" Jul 24 23:07:18.871017 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.870986 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kp6z4_b5f35b73-7542-425f-b5b3-1316baa372d8/kube-multus/0.log" Jul 24 23:07:18.959764 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.959730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t42k6_a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b/network-metrics-daemon/0.log" Jul 24 23:07:18.971355 ip-10-0-139-136 kubenswrapper[2576]: I0724 23:07:18.971331 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t42k6_a0cedf5c-1a7f-4b63-8797-f6a4f8e13c6b/kube-rbac-proxy/0.log"