Jul 02 23:22:25.779939 ip-10-0-130-115 systemd[1]: Starting Kubernetes Kubelet... Jul 02 23:22:26.200539 ip-10-0-130-115 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:22:26.200539 ip-10-0-130-115 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 23:22:26.200539 ip-10-0-130-115 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:22:26.200539 ip-10-0-130-115 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 23:22:26.200539 ip-10-0-130-115 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:22:26.201485 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.201354 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 23:22:26.205706 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205691 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.205706 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205706 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205711 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205715 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205717 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205720 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205723 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205726 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205729 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205732 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205742 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205744 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205747 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205750 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205752 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205757 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205761 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205764 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205767 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205770 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205773 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.205765 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205775 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205778 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205781 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205783 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205786 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205788 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205791 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205793 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205796 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205799 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205801 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205804 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205807 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205809 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205811 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205814 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205818 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205822 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205826 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.206231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205829 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205832 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205834 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205837 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205839 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205841 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205844 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205846 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205848 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205851 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205853 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205855 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205858 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205860 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205870 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205874 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205877 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205880 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205882 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205885 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.206710 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205888 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205891 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205893 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205895 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205898 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205900 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205902 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205905 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205907 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205909 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205912 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205915 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205919 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205921 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205924 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205926 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205929 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205931 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205933 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205936 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.207227 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205938 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205941 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205943 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205946 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205948 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205950 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205953 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205956 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205959 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205961 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205964 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205966 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205969 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205971 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205974 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205976 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205978 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205981 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.205983 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206693 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.207723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206700 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206703 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206706 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206710 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206713 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206717 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206720 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206723 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206726 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206729 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206732 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206734 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206737 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206740 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206742 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206745 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206747 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206750 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206753 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206757 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.208226 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206759 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206762 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206765 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206768 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206772 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206776 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206779 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206782 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206785 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206787 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206790 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206793 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206796 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206799 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206801 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206804 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206808 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206812 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206815 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.208730 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206819 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206821 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206824 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206827 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206830 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206832 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206835 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206838 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206841 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206843 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206846 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206849 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206852 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206854 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206857 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206859 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206862 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206865 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206867 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206870 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.209211 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206875 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206878 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206880 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206883 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206885 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206888 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206890 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206892 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206895 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206897 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206900 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206903 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206906 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206909 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206911 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206914 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206916 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206919 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206921 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206924 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.209707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206926 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206929 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206931 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206934 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206936 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206939 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206941 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206944 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206947 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206949 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206951 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206954 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206957 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206959 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206962 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206964 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206967 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206970 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.206973 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207355 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.210201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207360 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207362 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207365 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207368 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207371 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207374 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207376 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207379 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207381 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207383 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207386 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207388 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207391 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207393 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207396 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207399 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207401 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207404 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207407 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207409 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.210707 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207412 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207414 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207417 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207419 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207421 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207424 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207427 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207431 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207434 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207436 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207439 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207441 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207444 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207446 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207448 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207451 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207454 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207457 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207459 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.211191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207462 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207464 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207467 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207469 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207472 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207503 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207509 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207514 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207519 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207523 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207527 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207532 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207535 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207539 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207542 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207545 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207547 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207550 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207553 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.211672 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207555 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207558 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207560 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207562 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207565 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207567 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207570 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207572 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207574 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207577 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207581 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207587 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207590 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207592 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207595 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207598 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207601 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207604 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207606 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207609 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.212223 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207611 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207614 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207618 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207620 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207623 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207625 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207628 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207630 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207632 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207635 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207637 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207640 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207642 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207645 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207647 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207649 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207652 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207654 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207657 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.207659 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.212728 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207728 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207738 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207757 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207765 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207770 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207774 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207778 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207782 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207785 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207788 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207791 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207794 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207797 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207800 2561 flags.go:64] FLAG: --cgroup-root="" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207803 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207806 2561 flags.go:64] FLAG: --client-ca-file="" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207808 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207811 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207815 2561 flags.go:64] FLAG: --cluster-domain="" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207818 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207821 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207824 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207827 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207831 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 23:22:26.213197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207833 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207836 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207840 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207843 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207845 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207848 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207851 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207854 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207862 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207865 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207868 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207873 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207876 2561 flags.go:64] FLAG: --enable-server="true" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207879 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207883 2561 flags.go:64] FLAG: --event-burst="100" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207887 2561 flags.go:64] FLAG: --event-qps="50" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207889 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207892 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207895 2561 flags.go:64] FLAG: --eviction-hard="" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207898 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207901 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207904 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207907 2561 flags.go:64] FLAG: --eviction-soft="" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207909 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207912 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 23:22:26.213778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207915 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207918 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207920 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207923 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207926 2561 flags.go:64] FLAG: --feature-gates="" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207929 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207932 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207935 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207938 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207941 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207943 2561 flags.go:64] FLAG: --help="false" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207946 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207951 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207954 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207957 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207960 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207963 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207966 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207970 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207973 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207976 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207980 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207983 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207985 2561 flags.go:64] FLAG: --kube-reserved="" Jul 02 23:22:26.214444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207988 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207990 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207993 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207996 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.207998 2561 flags.go:64] FLAG: --lock-file="" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208001 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208003 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208006 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208012 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208014 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208017 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208020 2561 flags.go:64] FLAG: --logging-format="text" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208022 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208025 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208028 2561 flags.go:64] FLAG: --manifest-url="" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208031 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208035 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208038 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208041 2561 flags.go:64] FLAG: --max-pods="110" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208044 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208048 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208051 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208054 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208057 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208059 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 23:22:26.215027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208062 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208070 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208073 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208076 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208079 2561 flags.go:64] FLAG: --pod-cidr="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208082 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208087 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208090 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208093 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208096 2561 flags.go:64] FLAG: --port="10250" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208099 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208102 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-023439e5d1529af0e" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208105 2561 flags.go:64] FLAG: --qos-reserved="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208108 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208111 2561 flags.go:64] FLAG: --register-node="true" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208114 2561 flags.go:64] FLAG: --register-with-taints="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208117 2561 flags.go:64] FLAG: --registry-burst="10" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208120 2561 flags.go:64] FLAG: --registry-qps="5" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208122 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208125 2561 flags.go:64] FLAG: --reserved-memory="" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208129 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208132 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208135 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208137 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208140 2561 flags.go:64] FLAG: --runonce="false" Jul 02 23:22:26.215655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208143 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208146 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208149 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208153 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208156 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208159 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208162 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208165 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208168 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208171 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208174 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208177 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208180 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208183 2561 flags.go:64] FLAG: --system-cgroups="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208185 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208190 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208193 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208196 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208200 2561 flags.go:64] FLAG: --tls-min-version="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208203 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208206 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208208 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208211 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208214 2561 flags.go:64] FLAG: --v="2" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208217 2561 flags.go:64] FLAG: --version="false" Jul 02 23:22:26.216264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208221 2561 flags.go:64] FLAG: --vmodule="" Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208224 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208227 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208350 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208355 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208358 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208361 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208363 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208366 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208368 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208373 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208376 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208379 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208383 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208387 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208391 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208394 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208397 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208400 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.216871 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208402 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208405 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208408 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208411 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208413 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208416 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208418 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208421 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208423 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208426 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208429 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208431 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208433 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208436 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208438 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208441 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208444 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208446 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208449 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208452 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.217349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208454 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208457 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208459 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208463 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208466 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208469 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208471 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208473 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208477 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208480 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208483 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208485 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208488 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208490 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208493 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208496 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208500 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208503 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208506 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.217830 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208509 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208512 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208514 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208517 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208520 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208522 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208525 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208528 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208530 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208533 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208535 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208539 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208542 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208545 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208548 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208551 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208555 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208558 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208560 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.218346 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208563 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208565 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208569 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208572 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208574 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208577 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208579 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208582 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208584 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208587 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208590 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208592 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208595 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208597 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208600 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208602 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208605 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208607 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208610 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208612 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.218806 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208615 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208617 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208620 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208622 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.208625 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.208631 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.215559 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.215574 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.216334 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.216341 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216422 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216426 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216430 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216433 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216436 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.219279 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216439 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216441 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216444 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216446 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216450 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216455 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216459 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216462 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216465 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216468 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216471 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216474 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216476 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216479 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216481 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216484 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216487 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216490 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216492 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.219655 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216495 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216497 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216500 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216502 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216504 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216507 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216510 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216512 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216514 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216517 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216519 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216521 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216524 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216526 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216528 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216531 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216533 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216536 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216538 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216540 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.220108 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216543 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216545 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216547 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216550 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216553 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216555 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216558 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216560 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216562 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216565 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216567 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216570 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216573 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216576 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216580 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216582 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216585 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216588 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216591 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.220597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216594 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216596 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216599 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216601 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216604 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216606 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216609 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216611 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216614 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216616 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216618 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216621 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216623 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216625 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216628 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216630 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216633 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216635 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216637 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216639 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.221040 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216642 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216644 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216647 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216649 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216651 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216654 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216656 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216658 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216661 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216663 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216666 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216668 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216671 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216673 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216675 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.221528 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216678 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.216683 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216802 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216807 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216809 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216812 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216815 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216818 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216820 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216823 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216825 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216828 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216830 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216834 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:22:26.221885 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216838 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216840 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216843 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216846 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216849 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216852 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216854 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216857 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216859 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216861 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216864 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216866 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216869 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216871 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216874 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216878 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216881 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216884 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216887 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216889 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:22:26.222224 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216892 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216894 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216896 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216899 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216901 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216904 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216906 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216908 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216911 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216913 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216916 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216918 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216920 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216923 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216925 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216927 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216930 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216932 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216935 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216937 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:22:26.222753 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216940 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216942 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216944 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216947 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216949 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216952 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216954 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216956 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216959 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216962 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216965 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216967 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216970 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216972 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216974 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216977 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216979 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216982 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216984 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216986 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:22:26.223231 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216989 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216991 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216994 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.216996 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217001 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217003 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217006 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217009 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217012 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217016 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217018 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217021 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217023 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217026 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217029 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217032 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217034 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217038 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217040 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:22:26.223723 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217043 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217046 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217049 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217052 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217054 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217057 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217059 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:26.217062 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.217067 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.217158 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.219217 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.220613 2561 server.go:1019] "Starting client certificate rotation" Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.220714 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 23:22:26.224466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.220757 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 23:22:26.224795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.223080 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 23:22:26.227304 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.227266 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 23:22:26.239544 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.239528 2561 log.go:25] "Validated CRI v1 runtime API" Jul 02 23:22:26.243275 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.243254 2561 log.go:25] "Validated CRI v1 image API" Jul 02 23:22:26.244137 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.244124 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 23:22:26.246561 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.246539 2561 fs.go:135] Filesystem UUIDs: map[00705c2f-bf19-4b57-a3f2-5a214d8391c3:/dev/nvme0n1p3 2a7c441e-aaa1-4676-bd79-618db3c48b88:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 02 23:22:26.246633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.246561 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 23:22:26.251766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.251657 2561 manager.go:216] Machine: {Timestamp:2026-07-02 23:22:26.250279049 +0000 UTC m=+0.332602963 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096606 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec296e75a1cb8c42e6088c94d25f5e26 SystemUUID:ec296e75-a1cb-8c42-e608-8c94d25f5e26 BootID:ca923c5f-35d4-43ac-a79c-91394456bb4b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d4:ff:6a:44:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d4:ff:6a:44:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:2f:dc:af:88:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 23:22:26.251766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.251758 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 23:22:26.251926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.251857 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 23:22:26.255336 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.255309 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 23:22:26.255472 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.255337 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 23:22:26.256227 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.256218 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 23:22:26.256270 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.256230 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 23:22:26.256270 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.256238 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 23:22:26.256938 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.256928 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 23:22:26.258528 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.258518 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 23:22:26.261524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.261513 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 02 23:22:26.261634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.261625 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 23:22:26.262331 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262322 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 02 23:22:26.262372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262336 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 23:22:26.262372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262351 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 23:22:26.262372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262361 2561 kubelet.go:398] "Adding apiserver pod source" Jul 02 23:22:26.262372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262369 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 23:22:26.262733 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262721 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 23:22:26.262763 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.262742 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 23:22:26.264984 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.264967 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 23:22:26.266953 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.266940 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 23:22:26.267019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.266969 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 23:22:26.268228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268218 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 23:22:26.268300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268233 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 23:22:26.268300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268239 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 23:22:26.268300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268244 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 23:22:26.268300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268252 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 23:22:26.268300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268257 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268317 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268324 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268330 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268335 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268347 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 23:22:26.268431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.268356 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 23:22:26.269219 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.269210 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 23:22:26.269219 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.269218 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 23:22:26.272276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.272261 2561 server.go:1268] "Started kubelet" Jul 02 23:22:26.272382 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.272340 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 23:22:26.276373 ip-10-0-130-115 systemd[1]: Started Kubernetes Kubelet. Jul 02 23:22:26.276676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.276646 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 23:22:26.276766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.276708 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 23:22:26.279303 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.279027 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 02 23:22:26.279784 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.279755 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 23:22:26.279953 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.279849 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 23:22:26.280210 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.280187 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 23:22:26.280833 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.280815 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 23:22:26.282969 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.282923 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 23:22:26.285821 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.284956 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7910ce6c8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.272234637 +0000 UTC m=+0.354558552,LastTimestamp:2026-07-02 23:22:26.272234637 +0000 UTC m=+0.354558552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.287410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.287388 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 23:22:26.287410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.287400 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 23:22:26.287575 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.287555 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.288266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288247 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 23:22:26.288266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288250 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 23:22:26.288430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288277 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 23:22:26.288430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288362 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 02 23:22:26.288430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288373 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 02 23:22:26.288513 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288483 2561 factory.go:153] Registering CRI-O factory Jul 02 23:22:26.288513 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288493 2561 factory.go:223] Registration of the crio container factory successfully Jul 02 23:22:26.288580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288531 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 23:22:26.288580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288538 2561 factory.go:55] Registering systemd factory Jul 02 23:22:26.288580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288542 2561 factory.go:223] Registration of the systemd container factory successfully Jul 02 23:22:26.288580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288557 2561 factory.go:103] Registering Raw factory Jul 02 23:22:26.288580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.288571 2561 manager.go:1196] Started watching for new ooms in manager Jul 02 23:22:26.289157 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.289144 2561 manager.go:349] Starting recovery of all containers Jul 02 23:22:26.292391 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.292367 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 23:22:26.298623 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.298605 2561 manager.go:354] Recovery completed Jul 02 23:22:26.299306 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.299262 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 23:22:26.299537 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.299383 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 23:22:26.302842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.302831 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.304997 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.304972 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.305070 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305009 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.305070 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305021 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.305605 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305589 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 23:22:26.305605 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305600 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.305722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305602 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 23:22:26.305722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.305637 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 02 23:22:26.306512 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.306491 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.306602 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.306534 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.306602 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.306550 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.306917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.306901 2561 policy_none.go:49] "None policy: Start" Jul 02 23:22:26.307009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.307000 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 23:22:26.307088 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.307079 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 23:22:26.308546 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.308463 2561 policy_none.go:47] "Start" Jul 02 23:22:26.313989 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.313920 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c25501 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.304996609 +0000 UTC m=+0.387320523,LastTimestamp:2026-07-02 23:22:26.304996609 +0000 UTC m=+0.387320523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.318655 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.318599 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c298f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305014006 +0000 UTC m=+0.387337920,LastTimestamp:2026-07-02 23:22:26.305014006 +0000 UTC m=+0.387337920,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.325526 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.325463 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c2cdee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305027566 +0000 UTC m=+0.387351483,LastTimestamp:2026-07-02 23:22:26.305027566 +0000 UTC m=+0.387351483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.337077 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.337017 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c25501\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c25501 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.304996609 +0000 UTC m=+0.387320523,LastTimestamp:2026-07-02 23:22:26.306511499 +0000 UTC m=+0.388835417,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.346817 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.346749 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c298f6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c298f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305014006 +0000 UTC m=+0.387337920,LastTimestamp:2026-07-02 23:22:26.306541374 +0000 UTC m=+0.388865290,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.354258 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354243 2561 manager.go:337] "Starting Device Plugin manager" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.354308 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354322 2561 server.go:87] "Starting device plugin registration server" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354580 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354592 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354684 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354761 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.354769 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.355307 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.355367 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.355936 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c2cdee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c2cdee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305027566 +0000 UTC m=+0.387351483,LastTimestamp:2026-07-02 23:22:26.306555461 +0000 UTC m=+0.388879378,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.365824 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.364383 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7915d4b6e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.356532961 +0000 UTC m=+0.438856863,LastTimestamp:2026-07-02 23:22:26.356532961 +0000 UTC m=+0.438856863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.437154 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.437113 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 23:22:26.438514 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.438495 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 23:22:26.438592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.438546 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 23:22:26.438592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.438578 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 23:22:26.438671 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.438619 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 23:22:26.447951 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.447925 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 02 23:22:26.455110 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.455071 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.455968 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.455954 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.456023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.455985 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.456023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.455998 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.456023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.456019 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.463757 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.463670 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c25501\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c25501 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.304996609 +0000 UTC m=+0.387320523,LastTimestamp:2026-07-02 23:22:26.455969056 +0000 UTC m=+0.538292971,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.474259 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.474235 2561 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.474340 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.474234 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c298f6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c298f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305014006 +0000 UTC m=+0.387337920,LastTimestamp:2026-07-02 23:22:26.455992649 +0000 UTC m=+0.538316564,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.477505 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.477489 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fj9b" Jul 02 23:22:26.484978 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.484921 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-115.ec2.internal.18be9c7912c2cdee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18be9c7912c2cdee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-02 23:22:26.305027566 +0000 UTC m=+0.387351483,LastTimestamp:2026-07-02 23:22:26.456002302 +0000 UTC m=+0.538326217,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}" Jul 02 23:22:26.486521 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.486506 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fj9b" Jul 02 23:22:26.511013 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.510994 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.539134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.539108 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"] Jul 02 23:22:26.539231 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.539197 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.542873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.542860 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.542957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.542891 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.542957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.542903 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.544372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.544359 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.544503 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.544485 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.544542 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.544520 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.545143 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545127 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.545206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545154 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.545206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545132 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.545206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545164 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.545206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545179 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.545206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.545202 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.546423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.546407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.546517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.546432 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.547133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.547116 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.547212 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.547146 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.547212 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.547159 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.561804 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.561786 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.565163 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.565146 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.590040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.590020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.590117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.590044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.590117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.590061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.675169 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.675138 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:26.676215 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.676199 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:26.676319 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.676231 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:26.676319 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.676244 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:26.676319 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.676269 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.682621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.682601 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.682702 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.682634 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-115.ec2.internal\": node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.690710 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690691 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.690789 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690717 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.690789 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.690876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690778 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.690876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.690876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.690817 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.695696 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.695678 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.796023 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.795961 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.865186 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.865148 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.870493 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:26.869378 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:26.896571 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.896541 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:26.997113 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:26.997074 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:27.097593 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:27.097520 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:27.184714 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.184686 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:22:27.198056 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:27.198033 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:27.220216 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.220190 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 23:22:27.220742 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.220351 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 23:22:27.298429 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:27.298392 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:27.336017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.335990 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:22:27.346802 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.346778 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:22:27.348099 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.348079 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:22:27.348183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.348117 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:22:27.348183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.348134 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:22:27.357697 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:27.357668 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b46bf19a4c8c74cc56a4f755de60986.slice/crio-f950f300a08494d197fbe12dc40ec7afa0120eb731e5b110b17ae88857286bbc WatchSource:0}: Error finding container f950f300a08494d197fbe12dc40ec7afa0120eb731e5b110b17ae88857286bbc: Status 404 returned error can't find the container with id f950f300a08494d197fbe12dc40ec7afa0120eb731e5b110b17ae88857286bbc Jul 02 23:22:27.363820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.363803 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:22:27.399212 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:27.399191 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found" Jul 02 23:22:27.433237 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.433218 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:22:27.442106 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.442023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerStarted","Data":"b91f69d522504574e33f9dce80b8d702113a45de7d6455d015e58244f532ab6b"} Jul 02 23:22:27.442903 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.442876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" event={"ID":"4b46bf19a4c8c74cc56a4f755de60986","Type":"ContainerStarted","Data":"f950f300a08494d197fbe12dc40ec7afa0120eb731e5b110b17ae88857286bbc"} Jul 02 23:22:27.488133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.488102 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" Jul 02 23:22:27.488222 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.488119 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 23:17:27 +0000 UTC" deadline="2028-03-10 14:59:18.976805978 +0000 UTC" Jul 02 23:22:27.488222 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.488149 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14799h36m51.488660595s" Jul 02 23:22:27.498189 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.498168 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 23:22:27.499098 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.499085 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" Jul 02 23:22:27.507388 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.507373 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 23:22:27.915561 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:27.915529 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:22:28.264069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.264038 2561 apiserver.go:52] "Watching apiserver" Jul 02 23:22:28.272928 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.272904 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 23:22:28.273349 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.273297 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2rngq","openshift-image-registry/node-ca-q2hx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-kwsbr","openshift-multus/multus-fztcg","openshift-multus/network-metrics-daemon-59s58","kube-system/konnectivity-agent-dpjvf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj","openshift-dns/node-resolver-pq92b","openshift-network-diagnostics/network-check-target-bwtfb","openshift-network-operator/iptables-alerter-wfv9z","openshift-ovn-kubernetes/ovnkube-node-qqq9g","kube-system/global-pull-secret-syncer-rx9mk"] Jul 02 23:22:28.275189 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.275166 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.277407 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.277215 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.278906 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.278888 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.279077 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.279057 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mfnms\"" Jul 02 23:22:28.279401 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.279382 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.280059 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.280036 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.280059 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.280051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.280708 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.280684 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 23:22:28.280969 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.280926 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:28.281152 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.281133 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gbb7s\"" Jul 02 23:22:28.281701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.281673 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.282096 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.282385 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.282498 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.282819 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.283379 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 23:22:28.283553 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.283487 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qksbw\"" Jul 02 23:22:28.283881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.283720 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.284236 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.284209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.285156 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.285133 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.286198 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.286177 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.286761 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.286741 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 23:22:28.287145 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.287123 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.287388 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.287349 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 23:22:28.287490 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.287474 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.288454 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.288433 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 23:22:28.288562 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.288538 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 23:22:28.289383 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.289350 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 23:22:28.289500 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.289427 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjhb9\"" Jul 02 23:22:28.289606 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.289577 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xdq4h\"" Jul 02 23:22:28.289796 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.289778 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.290105 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.289871 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.290223 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.290196 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:28.290546 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.290525 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 23:22:28.290665 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.290649 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.290815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.290797 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.291132 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.291114 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g47tw\"" Jul 02 23:22:28.291236 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.291206 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.293599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.293580 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.293696 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.293645 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4rgtx\"" Jul 02 23:22:28.293759 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.293646 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.294078 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.294021 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.295401 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.295353 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:28.295524 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.295501 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:28.296549 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.296525 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.296646 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.296562 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 23:22:28.297203 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.297179 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rbs4z\"" Jul 02 23:22:28.297320 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.297188 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 23:22:28.298989 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.298964 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:22:28.299279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.299255 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 23:22:28.299399 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.299349 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftggf\"" Jul 02 23:22:28.299544 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.299525 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 23:22:28.301315 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301284 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 23:22:28.301459 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301430 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2754126-6ebb-4f43-8ede-c67a663747a9-konnectivity-ca\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.301537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovn-node-metrics-cert\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.301600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301533 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-netns\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301566 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-bin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301596 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-multus\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301623 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-os-release\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301649 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-kubelet\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301698 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-cnibin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-hostroot\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301792 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-system-cni-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301850 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.301893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301913 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-ovn\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-log-socket\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.301976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqhg\" (UniqueName: \"kubernetes.io/projected/361985eb-e209-4f32-8465-45c2d2528cb3-kube-api-access-zhqhg\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-host\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302029 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2754126-6ebb-4f43-8ede-c67a663747a9-agent-certs\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-kubelet\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302106 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-systemd-units\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302132 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-script-lib\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302157 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-lib-modules\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302202 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-tmp\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302241 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2091d71-7dcc-43ad-807d-3ded4bf54cec-host\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302307 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdv8\" (UniqueName: \"kubernetes.io/projected/e24fae4e-4711-4b22-bd1e-066b882107a3-kube-api-access-mvdv8\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-etc-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302383 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-socket-dir-parent\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302411 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-multus-daemon-config\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302434 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcb7g\" (UniqueName: \"kubernetes.io/projected/412e9e17-1969-40a1-829b-79346d9d4d34-kube-api-access-dcb7g\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302459 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-modprobe-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302482 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d9db1-dbe0-4358-a3bb-315aac8af83a-tmp-dir\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5sd\" (UniqueName: \"kubernetes.io/projected/ab9f24c8-020e-448f-b0e6-a3c21338b478-kube-api-access-sb5sd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-kubernetes\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-var-lib-kubelet\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302611 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2091d71-7dcc-43ad-807d-3ded4bf54cec-serviceca\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302648 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-netns\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-env-overrides\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.302701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302689 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-etc-kubernetes\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6x24\" (UniqueName: \"kubernetes.io/projected/e2091d71-7dcc-43ad-807d-3ded4bf54cec-kube-api-access-g6x24\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302744 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrv4\" (UniqueName: \"kubernetes.io/projected/ace55929-22cf-4c09-b937-9187e68e79e3-kube-api-access-7jrv4\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302759 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-systemd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-netd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-dbus\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-conf\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-sys\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkczx\" (UniqueName: \"kubernetes.io/projected/119d9db1-dbe0-4358-a3bb-315aac8af83a-kube-api-access-vkczx\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302923 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302939 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-var-lib-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302953 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-bin\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.302993 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-multus-certs\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-run\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303051 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-os-release\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.303343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303067 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-socket-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303082 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119d9db1-dbe0-4358-a3bb-315aac8af83a-hosts-file\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303097 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303114 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-node-log\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-k8s-cni-cncf-io\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-conf-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303268 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-systemd\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303334 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303359 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysconfig\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303374 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-tuned\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-cnibin\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303409 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-registration-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-sys-fs\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-slash\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303467 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.303957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303483 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-kubelet-config\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.304486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-cni-binary-copy\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.304486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dp74\" (UniqueName: \"kubernetes.io/projected/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-kube-api-access-7dp74\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.304486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303531 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-device-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.304486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-config\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.304486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.303603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-system-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.323223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.323198 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl69j" Jul 02 23:22:28.331219 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.331138 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl69j" Jul 02 23:22:28.389918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.389891 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 23:22:28.404563 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404526 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-os-release\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.404563 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-socket-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404580 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-host-slash\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119d9db1-dbe0-4358-a3bb-315aac8af83a-hosts-file\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-node-log\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404686 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-os-release\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404697 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119d9db1-dbe0-4358-a3bb-315aac8af83a-hosts-file\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-k8s-cni-cncf-io\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404730 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-k8s-cni-cncf-io\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.404740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404740 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-node-log\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-conf-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404751 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-socket-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404779 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-conf-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404782 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-systemd\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404831 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-systemd\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404857 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404859 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysconfig\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404896 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysconfig\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-tuned\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-cnibin\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-registration-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-sys-fs\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404981 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-cnibin\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.405086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404985 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-registration-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.404992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-slash\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405027 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-sys-fs\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405038 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-slash\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405064 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-ovn-kubernetes\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-kubelet-config\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-cni-binary-copy\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405112 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-kubelet-config\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dp74\" (UniqueName: \"kubernetes.io/projected/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-kube-api-access-7dp74\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-device-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405184 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-config\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405307 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-device-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-system-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405360 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405372 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-system-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.405898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405389 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-iptables-alerter-script\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405419 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wd4\" (UniqueName: \"kubernetes.io/projected/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-kube-api-access-p8wd4\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405448 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2754126-6ebb-4f43-8ede-c67a663747a9-konnectivity-ca\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovn-node-metrics-cert\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-netns\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-bin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405548 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-multus\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-os-release\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405583 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-kubelet\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405593 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-netns\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-cnibin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405689 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-multus\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405709 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-cnibin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405719 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-cni-dir\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-hostroot\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405742 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-os-release\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405758 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.406723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405765 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-kubelet\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405766 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-var-lib-cni-bin\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405785 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-system-cni-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405802 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-hostroot\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.405865 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405878 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-system-cni-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.405933 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ace55929-22cf-4c09-b937-9187e68e79e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.405937 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:28.905900408 +0000 UTC m=+2.988224310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406123 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-cni-binary-copy\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406126 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e24fae4e-4711-4b22-bd1e-066b882107a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-ovn\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-ovn\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-log-socket\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqhg\" (UniqueName: \"kubernetes.io/projected/361985eb-e209-4f32-8465-45c2d2528cb3-kube-api-access-zhqhg\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.407825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406236 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-log-socket\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-host\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406244 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-config\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406283 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-host\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406319 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2754126-6ebb-4f43-8ede-c67a663747a9-konnectivity-ca\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406399 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2754126-6ebb-4f43-8ede-c67a663747a9-agent-certs\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406422 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-kubelet\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-systemd-units\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-script-lib\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406493 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-kubelet\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406543 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-systemd-units\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406554 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-lib-modules\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-tmp\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406596 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2091d71-7dcc-43ad-807d-3ded4bf54cec-host\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406623 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.408766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdv8\" (UniqueName: \"kubernetes.io/projected/e24fae4e-4711-4b22-bd1e-066b882107a3-kube-api-access-mvdv8\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406660 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406671 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-etc-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-socket-dir-parent\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406704 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2091d71-7dcc-43ad-807d-3ded4bf54cec-host\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406748 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-multus-socket-dir-parent\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-multus-daemon-config\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406792 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-lib-modules\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406807 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcb7g\" (UniqueName: \"kubernetes.io/projected/412e9e17-1969-40a1-829b-79346d9d4d34-kube-api-access-dcb7g\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-modprobe-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406858 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d9db1-dbe0-4358-a3bb-315aac8af83a-tmp-dir\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5sd\" (UniqueName: \"kubernetes.io/projected/ab9f24c8-020e-448f-b0e6-a3c21338b478-kube-api-access-sb5sd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406908 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-kubernetes\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-var-lib-kubelet\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.406974 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2091d71-7dcc-43ad-807d-3ded4bf54cec-serviceca\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407000 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-netns\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-env-overrides\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-etc-kubernetes\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.409610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407112 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6x24\" (UniqueName: \"kubernetes.io/projected/e2091d71-7dcc-43ad-807d-3ded4bf54cec-kube-api-access-g6x24\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407138 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrv4\" (UniqueName: \"kubernetes.io/projected/ace55929-22cf-4c09-b937-9187e68e79e3-kube-api-access-7jrv4\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-systemd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407190 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-netd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-dbus\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407258 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-conf\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407287 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovnkube-script-lib\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-sys\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkczx\" (UniqueName: \"kubernetes.io/projected/119d9db1-dbe0-4358-a3bb-315aac8af83a-kube-api-access-vkczx\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-var-lib-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-bin\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407495 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-multus-certs\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407526 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-run\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407559 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-sys\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407627 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-run\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.410247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.407846 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-run-netns\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408001 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2091d71-7dcc-43ad-807d-3ded4bf54cec-serviceca\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408068 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-run-systemd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408104 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408116 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-netd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408154 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ab9f24c8-020e-448f-b0e6-a3c21338b478-env-overrides\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408199 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-etc-kubernetes\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408219 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-dbus\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.408303 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408340 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-kubernetes\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.408380 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:28.908339592 +0000 UTC m=+2.990663498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408390 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-etc-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408426 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-var-lib-kubelet\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408457 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-var-lib-openvswitch\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408479 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-sysctl-conf\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408479 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab9f24c8-020e-448f-b0e6-a3c21338b478-host-cni-bin\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408520 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/361985eb-e209-4f32-8465-45c2d2528cb3-host-run-multus-certs\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.411064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408594 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-modprobe-d\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.408908 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/361985eb-e209-4f32-8465-45c2d2528cb3-multus-daemon-config\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.409059 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d9db1-dbe0-4358-a3bb-315aac8af83a-tmp-dir\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.409122 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e24fae4e-4711-4b22-bd1e-066b882107a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.409851 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-tmp\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.409873 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-etc-tuned\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.411753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.409920 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab9f24c8-020e-448f-b0e6-a3c21338b478-ovn-node-metrics-cert\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.412537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.412514 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2754126-6ebb-4f43-8ede-c67a663747a9-agent-certs\") pod \"konnectivity-agent-dpjvf\" (UID: \"c2754126-6ebb-4f43-8ede-c67a663747a9\") " pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.414746 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.414679 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dp74\" (UniqueName: \"kubernetes.io/projected/bb2ef55f-dd53-43b3-8981-16fa7c83e40d-kube-api-access-7dp74\") pod \"tuned-2rngq\" (UID: \"bb2ef55f-dd53-43b3-8981-16fa7c83e40d\") " pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.417220 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.417185 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrv4\" (UniqueName: \"kubernetes.io/projected/ace55929-22cf-4c09-b937-9187e68e79e3-kube-api-access-7jrv4\") pod \"aws-ebs-csi-driver-node-gt5sj\" (UID: \"ace55929-22cf-4c09-b937-9187e68e79e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.417886 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.417861 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6x24\" (UniqueName: \"kubernetes.io/projected/e2091d71-7dcc-43ad-807d-3ded4bf54cec-kube-api-access-g6x24\") pod \"node-ca-q2hx8\" (UID: \"e2091d71-7dcc-43ad-807d-3ded4bf54cec\") " pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.418041 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.418014 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdv8\" (UniqueName: \"kubernetes.io/projected/e24fae4e-4711-4b22-bd1e-066b882107a3-kube-api-access-mvdv8\") pod \"multus-additional-cni-plugins-kwsbr\" (UID: \"e24fae4e-4711-4b22-bd1e-066b882107a3\") " pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.418684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.418661 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcb7g\" (UniqueName: \"kubernetes.io/projected/412e9e17-1969-40a1-829b-79346d9d4d34-kube-api-access-dcb7g\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.418775 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.418724 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqhg\" (UniqueName: \"kubernetes.io/projected/361985eb-e209-4f32-8465-45c2d2528cb3-kube-api-access-zhqhg\") pod \"multus-fztcg\" (UID: \"361985eb-e209-4f32-8465-45c2d2528cb3\") " pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.419004 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.418982 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkczx\" (UniqueName: \"kubernetes.io/projected/119d9db1-dbe0-4358-a3bb-315aac8af83a-kube-api-access-vkczx\") pod \"node-resolver-pq92b\" (UID: \"119d9db1-dbe0-4358-a3bb-315aac8af83a\") " pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.421095 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.421075 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5sd\" (UniqueName: \"kubernetes.io/projected/ab9f24c8-020e-448f-b0e6-a3c21338b478-kube-api-access-sb5sd\") pod \"ovnkube-node-qqq9g\" (UID: \"ab9f24c8-020e-448f-b0e6-a3c21338b478\") " pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.508017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.507922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-host-slash\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.508017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.507977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:28.508017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.508002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-iptables-alerter-script\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.508322 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.508026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wd4\" (UniqueName: \"kubernetes.io/projected/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-kube-api-access-p8wd4\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.508322 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.508063 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-host-slash\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.508638 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.508621 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-iptables-alerter-script\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.514721 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.514692 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:28.514721 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.514723 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:28.514895 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.514735 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:28.514895 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.514798 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:22:29.014780003 +0000 UTC m=+3.097103908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:28.516576 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.516553 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wd4\" (UniqueName: \"kubernetes.io/projected/85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0-kube-api-access-p8wd4\") pod \"iptables-alerter-wfv9z\" (UID: \"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0\") " pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.593611 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.593565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pq92b" Jul 02 23:22:28.603018 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.602988 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:28.613351 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.613323 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2hx8" Jul 02 23:22:28.621307 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.621283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" Jul 02 23:22:28.628151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.628132 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fztcg" Jul 02 23:22:28.635009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.634982 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" Jul 02 23:22:28.640964 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.640944 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rngq" Jul 02 23:22:28.647685 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.647665 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:28.656780 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.656757 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfv9z" Jul 02 23:22:28.912447 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.912367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:28.912701 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:28.912452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:28.912701 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.912508 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:28.912701 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.912573 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:29.912555596 +0000 UTC m=+3.994879506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:28.912701 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.912587 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:28.912701 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:28.912654 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:29.91263575 +0000 UTC m=+3.994959656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:29.113475 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.113431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:29.113671 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.113649 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:29.113732 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.113682 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:29.113732 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.113697 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:29.113835 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.113768 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:22:30.113747279 +0000 UTC m=+4.196071186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:29.333087 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.332963 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 23:17:29 +0000 UTC" deadline="2028-03-25 05:35:42.319600722 +0000 UTC" Jul 02 23:22:29.333087 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.333027 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15150h13m12.986576563s" Jul 02 23:22:29.397301 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.397266 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace55929_22cf_4c09_b937_9187e68e79e3.slice/crio-d45cd26f12cad6a9d04406f7bab8c887f0a5b5f695dd57cec249b7e12e63b61f WatchSource:0}: Error finding container d45cd26f12cad6a9d04406f7bab8c887f0a5b5f695dd57cec249b7e12e63b61f: Status 404 returned error can't find the container with id d45cd26f12cad6a9d04406f7bab8c887f0a5b5f695dd57cec249b7e12e63b61f Jul 02 23:22:29.400395 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.400371 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24fae4e_4711_4b22_bd1e_066b882107a3.slice/crio-15a3d0037838e9ce8fd47eaeb4dea8f4ae4aa3e48a0be3a414626e9ede714dac WatchSource:0}: Error finding container 15a3d0037838e9ce8fd47eaeb4dea8f4ae4aa3e48a0be3a414626e9ede714dac: Status 404 returned error can't find the container with id 15a3d0037838e9ce8fd47eaeb4dea8f4ae4aa3e48a0be3a414626e9ede714dac Jul 02 23:22:29.404659 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.404632 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361985eb_e209_4f32_8465_45c2d2528cb3.slice/crio-a77d5d74d4ca79906c2855ff6ff2c8b9f74aad640221903d6791480d1b55f9af WatchSource:0}: Error finding container a77d5d74d4ca79906c2855ff6ff2c8b9f74aad640221903d6791480d1b55f9af: Status 404 returned error can't find the container with id a77d5d74d4ca79906c2855ff6ff2c8b9f74aad640221903d6791480d1b55f9af Jul 02 23:22:29.405842 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.405815 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2754126_6ebb_4f43_8ede_c67a663747a9.slice/crio-66175a6dddd2b312c8874f94fcc60d570c2d15de6bb29f35a0bfe16fce01144e WatchSource:0}: Error finding container 66175a6dddd2b312c8874f94fcc60d570c2d15de6bb29f35a0bfe16fce01144e: Status 404 returned error can't find the container with id 66175a6dddd2b312c8874f94fcc60d570c2d15de6bb29f35a0bfe16fce01144e Jul 02 23:22:29.407657 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.407631 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119d9db1_dbe0_4358_a3bb_315aac8af83a.slice/crio-4a2b481b018af2908853f7af06f8491066607ce02644bf014b219403762ab060 WatchSource:0}: Error finding container 4a2b481b018af2908853f7af06f8491066607ce02644bf014b219403762ab060: Status 404 returned error can't find the container with id 4a2b481b018af2908853f7af06f8491066607ce02644bf014b219403762ab060 Jul 02 23:22:29.408852 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.408754 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2ef55f_dd53_43b3_8981_16fa7c83e40d.slice/crio-539d996da82b6b3d590cc329cdf5c9b50eeb0ffbeca748e3ccab9b99a2449d20 WatchSource:0}: Error finding container 539d996da82b6b3d590cc329cdf5c9b50eeb0ffbeca748e3ccab9b99a2449d20: Status 404 returned error can't find the container with id 539d996da82b6b3d590cc329cdf5c9b50eeb0ffbeca748e3ccab9b99a2449d20 Jul 02 23:22:29.410597 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.410551 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2091d71_7dcc_43ad_807d_3ded4bf54cec.slice/crio-ba86dd0cb17bfb0e88c0e7b29d5275dfdc6a98b1794a5cc947e8f376d4317b4c WatchSource:0}: Error finding container ba86dd0cb17bfb0e88c0e7b29d5275dfdc6a98b1794a5cc947e8f376d4317b4c: Status 404 returned error can't find the container with id ba86dd0cb17bfb0e88c0e7b29d5275dfdc6a98b1794a5cc947e8f376d4317b4c Jul 02 23:22:29.411271 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.411231 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9f24c8_020e_448f_b0e6_a3c21338b478.slice/crio-7326b6a1b88b49a40696f9160d79afa74bc591dbeb4e3ef21a66ce6227ede562 WatchSource:0}: Error finding container 7326b6a1b88b49a40696f9160d79afa74bc591dbeb4e3ef21a66ce6227ede562: Status 404 returned error can't find the container with id 7326b6a1b88b49a40696f9160d79afa74bc591dbeb4e3ef21a66ce6227ede562 Jul 02 23:22:29.414130 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:22:29.414103 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b53eb0_b156_4ab3_a8af_6f44aa2ae9e0.slice/crio-410cb60c4652a9729d7e02373b469d6be75a2e42c83f71214ef26263961ab6c4 WatchSource:0}: Error finding container 410cb60c4652a9729d7e02373b469d6be75a2e42c83f71214ef26263961ab6c4: Status 404 returned error can't find the container with id 410cb60c4652a9729d7e02373b469d6be75a2e42c83f71214ef26263961ab6c4 Jul 02 23:22:29.451199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.451013 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfv9z" event={"ID":"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0","Type":"ContainerStarted","Data":"410cb60c4652a9729d7e02373b469d6be75a2e42c83f71214ef26263961ab6c4"} Jul 02 23:22:29.453233 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.453202 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"7326b6a1b88b49a40696f9160d79afa74bc591dbeb4e3ef21a66ce6227ede562"} Jul 02 23:22:29.454927 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.454857 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rngq" event={"ID":"bb2ef55f-dd53-43b3-8981-16fa7c83e40d","Type":"ContainerStarted","Data":"539d996da82b6b3d590cc329cdf5c9b50eeb0ffbeca748e3ccab9b99a2449d20"} Jul 02 23:22:29.457293 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.457269 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pq92b" event={"ID":"119d9db1-dbe0-4358-a3bb-315aac8af83a","Type":"ContainerStarted","Data":"4a2b481b018af2908853f7af06f8491066607ce02644bf014b219403762ab060"} Jul 02 23:22:29.458488 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.458465 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dpjvf" event={"ID":"c2754126-6ebb-4f43-8ede-c67a663747a9","Type":"ContainerStarted","Data":"66175a6dddd2b312c8874f94fcc60d570c2d15de6bb29f35a0bfe16fce01144e"} Jul 02 23:22:29.459583 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.459552 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fztcg" event={"ID":"361985eb-e209-4f32-8465-45c2d2528cb3","Type":"ContainerStarted","Data":"a77d5d74d4ca79906c2855ff6ff2c8b9f74aad640221903d6791480d1b55f9af"} Jul 02 23:22:29.460678 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.460655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerStarted","Data":"15a3d0037838e9ce8fd47eaeb4dea8f4ae4aa3e48a0be3a414626e9ede714dac"} Jul 02 23:22:29.461731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.461709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2hx8" event={"ID":"e2091d71-7dcc-43ad-807d-3ded4bf54cec","Type":"ContainerStarted","Data":"ba86dd0cb17bfb0e88c0e7b29d5275dfdc6a98b1794a5cc947e8f376d4317b4c"} Jul 02 23:22:29.462863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.462843 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" event={"ID":"ace55929-22cf-4c09-b937-9187e68e79e3","Type":"ContainerStarted","Data":"d45cd26f12cad6a9d04406f7bab8c887f0a5b5f695dd57cec249b7e12e63b61f"} Jul 02 23:22:29.770744 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.770719 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:22:29.921658 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.921622 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:29.921815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:29.921687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:29.921874 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.921819 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:29.921938 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.921886 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:31.921865959 +0000 UTC m=+6.004189865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:29.922419 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.922306 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:29.922419 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:29.922381 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:31.922347547 +0000 UTC m=+6.004671453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:30.122995 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.122928 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:30.123202 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.123181 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:30.123272 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.123211 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:30.123272 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.123227 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:30.123399 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.123287 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:22:32.123268322 +0000 UTC m=+6.205592228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:30.333824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.333668 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 23:17:29 +0000 UTC" deadline="2028-01-26 17:43:36.75907767 +0000 UTC" Jul 02 23:22:30.333824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.333733 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13746h21m6.425348688s" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.441513 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.441651 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.442091 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.442190 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.442260 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:30.442421 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:30.442341 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:30.486101 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.485914 2561 generic.go:356] "Generic (PLEG): container finished" podID="fc08c5a47b8f2a9d0e8afb0797a72354" containerID="dd695edf250c68c800d2358fafa7448e18eb6cc4eba3def43843794dd80ab4d9" exitCode=0 Jul 02 23:22:30.486101 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.486021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerDied","Data":"dd695edf250c68c800d2358fafa7448e18eb6cc4eba3def43843794dd80ab4d9"} Jul 02 23:22:30.505492 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:30.505398 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" event={"ID":"4b46bf19a4c8c74cc56a4f755de60986","Type":"ContainerStarted","Data":"4ceca59bd095839f8823acf6fb0f4cf63466b27cef3bb912c5c5defe23eee32c"} Jul 02 23:22:31.513496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:31.513450 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerStarted","Data":"1d0284c0b3758664804a5bf1f7937f9b5422783117b7491cab5d3a5454c5a730"} Jul 02 23:22:31.528837 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:31.528280 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" podStartSLOduration=3.52825361 podStartE2EDuration="3.52825361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:30.521198313 +0000 UTC m=+4.603522242" watchObservedRunningTime="2026-07-02 23:22:31.52825361 +0000 UTC m=+5.610577538" Jul 02 23:22:31.940926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:31.940832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:31.941109 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:31.940928 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:31.941109 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:31.941057 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:31.941242 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:31.941123 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:35.941104568 +0000 UTC m=+10.023428478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:31.941579 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:31.941557 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:31.941673 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:31.941622 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:35.941602704 +0000 UTC m=+10.023926612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:32.143430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:32.142722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:32.143430 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.142926 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:32.143430 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.142948 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:32.143430 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.142962 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:32.143430 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.143029 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:22:36.143005712 +0000 UTC m=+10.225329645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:32.440529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:32.440446 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:32.440714 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.440593 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:32.441000 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:32.440978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:32.441114 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.441091 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:32.441174 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:32.441151 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:32.441252 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:32.441236 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:34.441223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:34.440486 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:34.441223 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:34.440622 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:34.441223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:34.441056 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:34.441223 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:34.441167 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:34.442198 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:34.442012 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:34.442198 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:34.442153 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:35.975210 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:35.975166 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:35.975764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:35.975252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:35.975764 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:35.975420 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:35.975764 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:35.975494 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:43.975472649 +0000 UTC m=+18.057796578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:35.975981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:35.975919 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:35.975981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:35.975978 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:22:43.975961459 +0000 UTC m=+18.058285369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:36.177329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:36.176955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:36.177329 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.177095 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:36.177329 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.177118 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:36.177329 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.177136 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:36.177329 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.177201 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:22:44.177181458 +0000 UTC m=+18.259505367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:36.440400 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:36.440302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:36.440566 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:36.440427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:36.440566 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:36.440462 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:36.440566 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.440476 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:36.440566 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.440540 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:36.440719 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:36.440642 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:38.439558 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:38.439514 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:38.439558 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:38.439553 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:38.440061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:38.439572 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:38.440061 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:38.439673 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:38.440061 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:38.439823 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:38.440061 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:38.439915 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:40.439051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:40.439068 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:40.439174 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:40.439195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:40.439269 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:40.439520 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:40.439402 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:42.439413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:42.439530 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:42.439427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:42.439621 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:42.439413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:42.440019 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:42.439699 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:44.041119 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.041077 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:44.041632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.041163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:44.041632 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.041243 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:44.041632 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.041270 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:44.041632 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.041340 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:00.041318458 +0000 UTC m=+34.123642361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:22:44.041632 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.041362 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:00.041350303 +0000 UTC m=+34.123674205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:22:44.243022 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.242980 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:44.243194 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.243167 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:22:44.243352 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.243202 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:22:44.243352 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.243216 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:44.243352 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.243296 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:23:00.243260934 +0000 UTC m=+34.325584839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:22:44.439690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.439604 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:44.439690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.439604 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:44.439690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:44.439604 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:44.439939 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.439763 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:44.439939 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.439832 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:44.439939 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:44.439917 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:46.439782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:46.439734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:46.440209 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:46.439841 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:46.440271 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:46.440228 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:46.440487 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:46.440331 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:46.440487 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:46.440391 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:46.440487 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:46.440458 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:48.439536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:48.439505 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:48.439536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:48.439517 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:48.440115 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:48.439634 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:48.440115 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:48.439687 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:48.440115 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:48.439723 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:48.440115 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:48.439773 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:49.554996 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.554750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"a18a3a04b6cc938a21c78a765a2b2da3c542a4739ae2583b0384b2ca37f2cc64"} Jul 02 23:22:49.555764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.555093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"b412db133959c3bfa0e58e627ab6f615c2a1fdcc07b93adca6d23723b2e7fd3f"} Jul 02 23:22:49.555764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.555146 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"ca1c823c76abb95bd8b34891723c4cd542d061588f85e54d6d3f43d963248eb8"} Jul 02 23:22:49.555764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.555195 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"1e70cfd7d2c7fc2eab1048b299331489c1ccf868ac4ee6f85f2f2bd4c97d788f"} Jul 02 23:22:49.556341 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.556316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rngq" event={"ID":"bb2ef55f-dd53-43b3-8981-16fa7c83e40d","Type":"ContainerStarted","Data":"bde4232951d0364c0f78167489dc6b3c81b2c7360a5fe1f43e612c5a3e6bb3a3"} Jul 02 23:22:49.558311 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.558098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pq92b" event={"ID":"119d9db1-dbe0-4358-a3bb-315aac8af83a","Type":"ContainerStarted","Data":"468095a153323a00f93371c3a930692106d5ab047611401a1689d65b715ab646"} Jul 02 23:22:49.560149 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.560108 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dpjvf" event={"ID":"c2754126-6ebb-4f43-8ede-c67a663747a9","Type":"ContainerStarted","Data":"89749d95534f8a15be14b55f50b3cecaf959064f6d8a229986f6a340f5f6ab4e"} Jul 02 23:22:49.561821 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.561796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fztcg" event={"ID":"361985eb-e209-4f32-8465-45c2d2528cb3","Type":"ContainerStarted","Data":"9d6a5a6210e165cce6ab9934eb91b562693e921dbce6a06fa93bc25757f5730d"} Jul 02 23:22:49.563230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.563208 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="a42fa9e5ac136243e783dbe3dd4862a09d813a0ab98d9c685a3da00a475d3ef4" exitCode=0 Jul 02 23:22:49.563365 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.563307 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"a42fa9e5ac136243e783dbe3dd4862a09d813a0ab98d9c685a3da00a475d3ef4"} Jul 02 23:22:49.567092 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.567071 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2hx8" event={"ID":"e2091d71-7dcc-43ad-807d-3ded4bf54cec","Type":"ContainerStarted","Data":"b9629daabe7cfb0c197ecaa2458287c3f7d224b1c988aabf6717cb64fb81c2ea"} Jul 02 23:22:49.569116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.569098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" event={"ID":"ace55929-22cf-4c09-b937-9187e68e79e3","Type":"ContainerStarted","Data":"f1017baa692b45e804b6668d319b4fa6ba7655d2b1796d0ff22e27fdb0b29827"} Jul 02 23:22:49.571811 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.571762 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2rngq" podStartSLOduration=3.030836222 podStartE2EDuration="22.5717452s" totalImagesPullingTime="19.540908978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:49.571184537 +0000 UTC m=+23.653508461" watchObservedRunningTime="2026-07-02 23:22:49.5717452 +0000 UTC m=+23.654069128" Jul 02 23:22:49.571973 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.571942 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" podStartSLOduration=21.571932927 podStartE2EDuration="21.571932927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:31.528719626 +0000 UTC m=+5.611043556" watchObservedRunningTime="2026-07-02 23:22:49.571932927 +0000 UTC m=+23.654256852" Jul 02 23:22:49.584253 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.584209 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dpjvf" podStartSLOduration=11.174233764 podStartE2EDuration="22.584192126s" totalImagesPullingTime="11.409958362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:49.584110759 +0000 UTC m=+23.666434684" watchObservedRunningTime="2026-07-02 23:22:49.584192126 +0000 UTC m=+23.666516054" Jul 02 23:22:49.601659 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.601621 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2hx8" podStartSLOduration=3.066570082 podStartE2EDuration="22.601605963s" totalImagesPullingTime="19.535035881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:49.601324295 +0000 UTC m=+23.683648218" watchObservedRunningTime="2026-07-02 23:22:49.601605963 +0000 UTC m=+23.683929888" Jul 02 23:22:49.644067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.644014 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fztcg" podStartSLOduration=3.048333928 podStartE2EDuration="22.643994207s" totalImagesPullingTime="19.595660279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:49.643722428 +0000 UTC m=+23.726046351" watchObservedRunningTime="2026-07-02 23:22:49.643994207 +0000 UTC m=+23.726318132" Jul 02 23:22:49.660593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:49.660550 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pq92b" podStartSLOduration=3.119721106 podStartE2EDuration="22.660536506s" totalImagesPullingTime="19.5408154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:49.659959599 +0000 UTC m=+23.742283524" watchObservedRunningTime="2026-07-02 23:22:49.660536506 +0000 UTC m=+23.742860429" Jul 02 23:22:50.250863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.250808 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 23:22:50.369342 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.369225 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T23:22:50.250827974Z","UUID":"f489f4e3-ba58-4ce3-8997-c6dd3a91a63a","Handler":null,"Name":"","Endpoint":""} Jul 02 23:22:50.371124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.371010 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 23:22:50.371124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.371047 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 23:22:50.439303 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.439254 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:50.439480 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:50.439400 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:50.439480 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.439254 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:50.439565 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:50.439526 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:50.439626 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.439579 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:50.439727 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:50.439707 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:50.572606 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.572574 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" event={"ID":"ace55929-22cf-4c09-b937-9187e68e79e3","Type":"ContainerStarted","Data":"3fa80a919941eba87d5dfd556adcaecfd007127293080dd66d01e6dce9b5f061"} Jul 02 23:22:50.573956 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.573927 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfv9z" event={"ID":"85b53eb0-b156-4ab3-a8af-6f44aa2ae9e0","Type":"ContainerStarted","Data":"d3f8df33c5de30a7c203306b298e1271e0a5e7a91af1d58f7bca08ad62163187"} Jul 02 23:22:50.576704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.576671 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"e48e219a3dddef71a934ba4783c718ebab90b787e7ef51d1330a85056a63d99d"} Jul 02 23:22:50.576795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.576730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"01483360c0d0c1dec813ae321e47719c0d206961de6a161a45113f3e3fc6210c"} Jul 02 23:22:50.591547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:50.591507 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wfv9z" podStartSLOduration=4.060285599 podStartE2EDuration="23.591496242s" totalImagesPullingTime="19.531210643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:50.5913865 +0000 UTC m=+24.673710425" watchObservedRunningTime="2026-07-02 23:22:50.591496242 +0000 UTC m=+24.673820165" Jul 02 23:22:51.241720 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:51.241681 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:51.242345 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:51.242329 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:51.578473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:51.578438 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:51.578921 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:51.578795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dpjvf" Jul 02 23:22:52.439570 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.439360 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:52.439756 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.439368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:52.439756 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:52.439667 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:52.439881 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:52.439761 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:52.439881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.439401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:52.439881 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:52.439864 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:52.584303 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.584254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"506653b7cf809e38c80c56aaf17c589f8a196e2e549b42a85370cd422257935b"} Jul 02 23:22:52.587067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.587033 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" event={"ID":"ace55929-22cf-4c09-b937-9187e68e79e3","Type":"ContainerStarted","Data":"513f86ebb4ed7615d7da8d170a867b051802815bb6d72efc7d69eaefb0c4883c"} Jul 02 23:22:52.604355 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:52.604276 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gt5sj" podStartSLOduration=3.719605643 podStartE2EDuration="25.604256972s" totalImagesPullingTime="21.884651329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:52.603227905 +0000 UTC m=+26.685551830" watchObservedRunningTime="2026-07-02 23:22:52.604256972 +0000 UTC m=+26.686580891" Jul 02 23:22:54.439858 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:54.439820 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:54.440547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:54.439865 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:54.440547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:54.439819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:54.440547 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:54.439967 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:54.440547 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:54.440051 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:54.440547 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:54.440108 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:55.596141 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.595888 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="638c988b90acfef90188a0fc4b6fef17f4197a9fabd637fa4c11861fb05a8214" exitCode=0 Jul 02 23:22:55.596817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.595973 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"638c988b90acfef90188a0fc4b6fef17f4197a9fabd637fa4c11861fb05a8214"} Jul 02 23:22:55.599815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.599795 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" event={"ID":"ab9f24c8-020e-448f-b0e6-a3c21338b478","Type":"ContainerStarted","Data":"474342daed61e5b1949d7eb42be47d5f6e1137b534d3b88582c014dc9a09c4c3"} Jul 02 23:22:55.600116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.600097 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:55.600219 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.600150 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:55.616386 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.616367 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:55.639940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:55.639900 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" podStartSLOduration=9.033392271 podStartE2EDuration="28.639887832s" totalImagesPullingTime="19.606495561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:22:55.638356016 +0000 UTC m=+29.720679940" watchObservedRunningTime="2026-07-02 23:22:55.639887832 +0000 UTC m=+29.722211759" Jul 02 23:22:56.440804 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.440774 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:56.441110 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.440866 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:56.441110 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.440892 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:56.441110 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.440905 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:56.441110 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.441010 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:56.441110 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.441098 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:56.602777 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.602740 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:56.618025 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.618000 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:22:56.712399 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.712333 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx9mk"] Jul 02 23:22:56.712526 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.712415 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:56.712526 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.712491 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:56.715754 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.715732 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-59s58"] Jul 02 23:22:56.715874 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.715837 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:56.715942 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.715920 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:56.716618 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.716597 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwtfb"] Jul 02 23:22:56.716749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:56.716665 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:56.716811 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:56.716740 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:57.606617 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:57.606585 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="0fad467b601bf3792e9b39dbf46851af17c580e91c5a4e6b0cab6e558cf4ab8d" exitCode=0 Jul 02 23:22:57.607017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:57.606673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"0fad467b601bf3792e9b39dbf46851af17c580e91c5a4e6b0cab6e558cf4ab8d"} Jul 02 23:22:58.440429 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:58.440402 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:22:58.440562 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:58.440414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:22:58.440562 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:58.440536 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:22:58.440673 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:58.440619 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:22:58.440673 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:58.440422 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:22:58.440770 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:22:58.440732 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:22:59.613119 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:59.613082 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="21e51f63b7d56e368d1e956faac4326eca3f1566cb23936268f20e25a2712f8b" exitCode=0 Jul 02 23:22:59.613768 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:22:59.613133 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"21e51f63b7d56e368d1e956faac4326eca3f1566cb23936268f20e25a2712f8b"} Jul 02 23:23:00.062278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.062242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:00.062485 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.062315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:00.062485 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.062413 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:23:00.062583 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.062491 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs podName:412e9e17-1969-40a1-829b-79346d9d4d34 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:32.062470382 +0000 UTC m=+66.144794286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs") pod "network-metrics-daemon-59s58" (UID: "412e9e17-1969-40a1-829b-79346d9d4d34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:23:00.062583 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.062417 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:23:00.062583 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.062545 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret podName:3729cf5d-1ad7-4025-bc82-9b93a24d1a44 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:32.062535961 +0000 UTC m=+66.144859866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret") pod "global-pull-secret-syncer-rx9mk" (UID: "3729cf5d-1ad7-4025-bc82-9b93a24d1a44") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:23:00.263979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.263945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:00.264159 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.264129 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:23:00.264159 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.264164 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:23:00.264347 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.264178 2561 projected.go:196] Error preparing data for projected volume kube-api-access-swqsl for pod openshift-network-diagnostics/network-check-target-bwtfb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:23:00.264347 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.264243 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl podName:52b04c71-9ea9-4794-91a6-6edb7cbd1d5c nodeName:}" failed. No retries permitted until 2026-07-02 23:23:32.264222955 +0000 UTC m=+66.346546874 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-swqsl" (UniqueName: "kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl") pod "network-check-target-bwtfb" (UID: "52b04c71-9ea9-4794-91a6-6edb7cbd1d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.439081 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.439098 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:00.439098 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.439196 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwtfb" podUID="52b04c71-9ea9-4794-91a6-6edb7cbd1d5c" Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.439266 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59s58" podUID="412e9e17-1969-40a1-829b-79346d9d4d34" Jul 02 23:23:00.439591 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:00.439368 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx9mk" podUID="3729cf5d-1ad7-4025-bc82-9b93a24d1a44" Jul 02 23:23:01.739868 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.739792 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeReady" Jul 02 23:23:01.740443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.739948 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 23:23:01.774032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.773998 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx"] Jul 02 23:23:01.775788 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.775772 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.777991 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.777816 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wjjdb"] Jul 02 23:23:01.779635 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.779613 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hjwk8"] Jul 02 23:23:01.779805 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.779775 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.780713 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.780692 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7w8v2\"" Jul 02 23:23:01.780816 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.780725 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 02 23:23:01.780866 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.780692 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.780973 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.780951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 02 23:23:01.781064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.780985 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.781697 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.781679 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x"] Jul 02 23:23:01.781827 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.781804 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.782310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.782274 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.782534 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.782516 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 02 23:23:01.782645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.782540 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.782645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.782583 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 02 23:23:01.782645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.782623 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tk9j2\"" Jul 02 23:23:01.783488 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.783458 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt"] Jul 02 23:23:01.783637 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.783607 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.786499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.785943 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dxzlm\"" Jul 02 23:23:01.786499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.786424 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 02 23:23:01.787171 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.786837 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.787171 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.787022 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.787171 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.787047 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 02 23:23:01.788681 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.788661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 02 23:23:01.788833 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.788750 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 02 23:23:01.789276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.789173 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:23:01.789276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.789337 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" Jul 02 23:23:01.791624 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.791604 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.791717 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.791646 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.792011 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.791993 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mhk8g\"" Jul 02 23:23:01.792682 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.792091 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xszr5\"" Jul 02 23:23:01.792682 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.792315 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.793328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.793310 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.794503 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.794486 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 02 23:23:01.798382 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.798189 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f9cdf7f67-8pwfb"] Jul 02 23:23:01.800328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.799088 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.800328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.799496 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 02 23:23:01.802061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.801477 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 23:23:01.802061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.801732 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j45fm\"" Jul 02 23:23:01.802061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.801923 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 23:23:01.803007 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.802989 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 23:23:01.803329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.803308 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466"] Jul 02 23:23:01.804049 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.804027 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.806398 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.805078 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx"] Jul 02 23:23:01.806398 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.805121 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt"] Jul 02 23:23:01.806398 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.805152 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk"] Jul 02 23:23:01.806398 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.805688 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.806691 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.806868 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.806964 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807075 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807081 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807138 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807438 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6"] Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807439 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ztq86\"" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.807517 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:01.808242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.808057 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 02 23:23:01.809272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.808364 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.809272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.808515 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.809429 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.809329 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d2w2\"" Jul 02 23:23:01.813587 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.813554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wjjdb"] Jul 02 23:23:01.814316 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814297 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x"] Jul 02 23:23:01.814481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814468 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hjwk8"] Jul 02 23:23:01.814644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814601 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:23:01.814769 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814655 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f9cdf7f67-8pwfb"] Jul 02 23:23:01.814769 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814689 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s5tnc"] Jul 02 23:23:01.815029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.813951 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" Jul 02 23:23:01.817335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.814057 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 23:23:01.817335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.817339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 02 23:23:01.817686 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.817669 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.818851 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.817702 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7pj66\"" Jul 02 23:23:01.819354 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.817901 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 02 23:23:01.819354 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.818011 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.819524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.818179 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7cbmj\"" Jul 02 23:23:01.819724 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.818230 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.820684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.820309 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.822792 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.822773 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2g5lx"] Jul 02 23:23:01.822895 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.822878 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:01.824752 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824731 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466"] Jul 02 23:23:01.824902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824785 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk"] Jul 02 23:23:01.824902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824799 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qpvbp\"" Jul 02 23:23:01.824902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824804 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6"] Jul 02 23:23:01.824902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824832 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:01.824902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.824845 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2g5lx"] Jul 02 23:23:01.825416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.825097 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 23:23:01.825780 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.825764 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 23:23:01.826330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.826312 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s5tnc"] Jul 02 23:23:01.827183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.827164 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 23:23:01.827850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.827832 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 23:23:01.828016 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.828004 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 23:23:01.828143 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.828129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-24d9j\"" Jul 02 23:23:01.875452 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a946af-8ab2-4774-a67e-5125b517005e-serving-cert\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.875557 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjzc\" (UniqueName: \"kubernetes.io/projected/49a946af-8ab2-4774-a67e-5125b517005e-kube-api-access-fsjzc\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.875557 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875518 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-tmp\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.875664 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-config\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.875715 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.875765 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875735 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-snapshots\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.875817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875765 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3548bc-f22e-4b6e-a727-a342457af286-serving-cert\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.875817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875797 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6162850d-acc9-41e3-83f0-2c324b0e9f94-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.875917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875825 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.875917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk97c\" (UniqueName: \"kubernetes.io/projected/ab3548bc-f22e-4b6e-a727-a342457af286-kube-api-access-zk97c\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.875917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.875917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875902 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5htsh\" (UniqueName: \"kubernetes.io/projected/6162850d-acc9-41e3-83f0-2c324b0e9f94-kube-api-access-5htsh\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.876088 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.876088 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstw9\" (UniqueName: \"kubernetes.io/projected/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-kube-api-access-zstw9\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.876088 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.875999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-trusted-ca\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.876088 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.876028 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.977328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-snapshots\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/09c6f89b-4757-4174-ace6-96c36a9e0706-kube-api-access-zntnn\") pod \"volume-data-source-validator-5d456fdbcd-x8bpt\" (UID: \"09c6f89b-4757-4174-ace6-96c36a9e0706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3548bc-f22e-4b6e-a727-a342457af286-serving-cert\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6162850d-acc9-41e3-83f0-2c324b0e9f94-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977467 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwqr\" (UniqueName: \"kubernetes.io/projected/60558dee-31d1-49d1-85ef-6ad6f53e142f-kube-api-access-xxwqr\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.977516 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-config\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977548 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977607 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lv7s\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977665 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977689 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhrj\" (UniqueName: \"kubernetes.io/projected/03794c72-91ad-4ff1-87e5-16e6520e708e-kube-api-access-cdhrj\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977713 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-tmp\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-config\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977761 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977797 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25b4864a-b323-4420-a61c-86f098df9474-config-volume\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977844 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.977917 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk97c\" (UniqueName: \"kubernetes.io/projected/ab3548bc-f22e-4b6e-a727-a342457af286-kube-api-access-zk97c\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5htsh\" (UniqueName: \"kubernetes.io/projected/6162850d-acc9-41e3-83f0-2c324b0e9f94-kube-api-access-5htsh\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstw9\" (UniqueName: \"kubernetes.io/projected/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-kube-api-access-zstw9\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.977992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b4864a-b323-4420-a61c-86f098df9474-tmp-dir\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.978020 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-snapshots\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.978021 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:01.978582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.978258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbt25\" (UniqueName: \"kubernetes.io/projected/25b4864a-b323-4420-a61c-86f098df9474-kube-api-access-mbt25\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:01.979063 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979105 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6162850d-acc9-41e3-83f0-2c324b0e9f94-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:01.979133 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.479113739 +0000 UTC m=+36.561437644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-trusted-ca\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzfq\" (UniqueName: \"kubernetes.io/projected/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-kube-api-access-zvzfq\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979324 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-config\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979450 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpff\" (UniqueName: \"kubernetes.io/projected/ae9e6769-4db6-494b-a2d5-24f04a403f84-kube-api-access-ptpff\") pod \"network-check-source-989f8b8dc-hw5k6\" (UID: \"ae9e6769-4db6-494b-a2d5-24f04a403f84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979472 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979480 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03794c72-91ad-4ff1-87e5-16e6520e708e-cert\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979505 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979615 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a946af-8ab2-4774-a67e-5125b517005e-tmp\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a946af-8ab2-4774-a67e-5125b517005e-serving-cert\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjzc\" (UniqueName: \"kubernetes.io/projected/49a946af-8ab2-4774-a67e-5125b517005e-kube-api-access-fsjzc\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmn4d\" (UniqueName: \"kubernetes.io/projected/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-kube-api-access-qmn4d\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:01.980142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979890 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979921 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49a946af-8ab2-4774-a67e-5125b517005e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-default-certificate\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.979952 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.980010 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-stats-auth\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.980016 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab3548bc-f22e-4b6e-a727-a342457af286-trusted-ca\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.980979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.980036 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:01.983271 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.983249 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49a946af-8ab2-4774-a67e-5125b517005e-serving-cert\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.983526 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.983508 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.983811 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.983793 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3548bc-f22e-4b6e-a727-a342457af286-serving-cert\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.987700 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.987678 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk97c\" (UniqueName: \"kubernetes.io/projected/ab3548bc-f22e-4b6e-a727-a342457af286-kube-api-access-zk97c\") pod \"console-operator-68b757fb67-hjwk8\" (UID: \"ab3548bc-f22e-4b6e-a727-a342457af286\") " pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:01.987955 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.987933 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstw9\" (UniqueName: \"kubernetes.io/projected/aa35770d-8b0e-42b8-bed0-b4f90a9eed3d-kube-api-access-zstw9\") pod \"kube-storage-version-migrator-operator-846bf668f7-vtf6x\" (UID: \"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:01.988147 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.988127 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjzc\" (UniqueName: \"kubernetes.io/projected/49a946af-8ab2-4774-a67e-5125b517005e-kube-api-access-fsjzc\") pod \"insights-operator-5c847dcfc4-wjjdb\" (UID: \"49a946af-8ab2-4774-a67e-5125b517005e\") " pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:01.988267 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:01.988249 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5htsh\" (UniqueName: \"kubernetes.io/projected/6162850d-acc9-41e3-83f0-2c324b0e9f94-kube-api-access-5htsh\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:02.081578 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081522 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25b4864a-b323-4420-a61c-86f098df9474-config-volume\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.081578 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081640 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b4864a-b323-4420-a61c-86f098df9474-tmp-dir\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbt25\" (UniqueName: \"kubernetes.io/projected/25b4864a-b323-4420-a61c-86f098df9474-kube-api-access-mbt25\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.081703 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzfq\" (UniqueName: \"kubernetes.io/projected/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-kube-api-access-zvzfq\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081756 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpff\" (UniqueName: \"kubernetes.io/projected/ae9e6769-4db6-494b-a2d5-24f04a403f84-kube-api-access-ptpff\") pod \"network-check-source-989f8b8dc-hw5k6\" (UID: \"ae9e6769-4db6-494b-a2d5-24f04a403f84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.081775 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.581755183 +0000 UTC m=+36.664079085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:02.081822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081812 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03794c72-91ad-4ff1-87e5-16e6520e708e-cert\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081891 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmn4d\" (UniqueName: \"kubernetes.io/projected/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-kube-api-access-qmn4d\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-default-certificate\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.081984 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-stats-auth\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/09c6f89b-4757-4174-ace6-96c36a9e0706-kube-api-access-zntnn\") pod \"volume-data-source-validator-5d456fdbcd-x8bpt\" (UID: \"09c6f89b-4757-4174-ace6-96c36a9e0706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.082053 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:02.082511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082069 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.083199 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.082120 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.582101445 +0000 UTC m=+36.664425348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:02.083199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwqr\" (UniqueName: \"kubernetes.io/projected/60558dee-31d1-49d1-85ef-6ad6f53e142f-kube-api-access-xxwqr\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.083199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.083199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082766 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-config\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.083199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.082823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.085465 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.085420 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.088686 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.088337 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03794c72-91ad-4ff1-87e5-16e6520e708e-cert\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:02.088686 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.088601 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b4864a-b323-4420-a61c-86f098df9474-tmp-dir\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.088834 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.088705 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25b4864a-b323-4420-a61c-86f098df9474-config-volume\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.088946 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.088928 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.089538 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.089513 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.090443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.089881 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.090443 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.090094 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:02.090443 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.090110 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:02.090443 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.090166 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.590148538 +0000 UTC m=+36.672472441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:02.090443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090393 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-config\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.090731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090540 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.090731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090581 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lv7s\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.090731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090634 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.090731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhrj\" (UniqueName: \"kubernetes.io/projected/03794c72-91ad-4ff1-87e5-16e6520e708e-kube-api-access-cdhrj\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:02.090731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.090708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.091013 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.090851 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:02.091013 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.090901 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.590886633 +0000 UTC m=+36.673210552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:02.091860 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.091834 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.091966 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.091861 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:02.591839181 +0000 UTC m=+36.674163103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:02.092413 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.092392 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-default-certificate\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.092671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.092651 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-stats-auth\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.093250 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.093228 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.095039 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.095019 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpff\" (UniqueName: \"kubernetes.io/projected/ae9e6769-4db6-494b-a2d5-24f04a403f84-kube-api-access-ptpff\") pod \"network-check-source-989f8b8dc-hw5k6\" (UID: \"ae9e6769-4db6-494b-a2d5-24f04a403f84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" Jul 02 23:23:02.097246 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.097222 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmn4d\" (UniqueName: \"kubernetes.io/projected/8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6-kube-api-access-qmn4d\") pod \"service-ca-operator-5d46d9ffd4-6vzdk\" (UID: \"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.097378 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.097354 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.098229 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.098199 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzfq\" (UniqueName: \"kubernetes.io/projected/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-kube-api-access-zvzfq\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:02.098229 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.098219 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwqr\" (UniqueName: \"kubernetes.io/projected/60558dee-31d1-49d1-85ef-6ad6f53e142f-kube-api-access-xxwqr\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.098791 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.098771 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbt25\" (UniqueName: \"kubernetes.io/projected/25b4864a-b323-4420-a61c-86f098df9474-kube-api-access-mbt25\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.099234 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.099215 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntnn\" (UniqueName: \"kubernetes.io/projected/09c6f89b-4757-4174-ace6-96c36a9e0706-kube-api-access-zntnn\") pod \"volume-data-source-validator-5d456fdbcd-x8bpt\" (UID: \"09c6f89b-4757-4174-ace6-96c36a9e0706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" Jul 02 23:23:02.100759 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.100736 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" Jul 02 23:23:02.106114 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.106096 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhrj\" (UniqueName: \"kubernetes.io/projected/03794c72-91ad-4ff1-87e5-16e6520e708e-kube-api-access-cdhrj\") pod \"ingress-canary-2g5lx\" (UID: \"03794c72-91ad-4ff1-87e5-16e6520e708e\") " pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:02.106191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.106107 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lv7s\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.118141 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.117924 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:02.128006 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.127984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" Jul 02 23:23:02.136388 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.136324 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" Jul 02 23:23:02.165054 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.164478 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" Jul 02 23:23:02.173565 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.172362 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" Jul 02 23:23:02.185845 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.185296 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2g5lx" Jul 02 23:23:02.337450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.335587 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wjjdb"] Jul 02 23:23:02.348621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.347509 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hjwk8"] Jul 02 23:23:02.371212 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.369821 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3548bc_f22e_4b6e_a727_a342457af286.slice/crio-0da97be55dbe02f88c120b034d75af35bf314d90e607c712b052195a196075c0 WatchSource:0}: Error finding container 0da97be55dbe02f88c120b034d75af35bf314d90e607c712b052195a196075c0: Status 404 returned error can't find the container with id 0da97be55dbe02f88c120b034d75af35bf314d90e607c712b052195a196075c0 Jul 02 23:23:02.379892 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.377469 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x"] Jul 02 23:23:02.379892 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.379474 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa35770d_8b0e_42b8_bed0_b4f90a9eed3d.slice/crio-1953472ca50d682768f1cdf279ac613d15a98d75bfa1ecba86ddea3734613fed WatchSource:0}: Error finding container 1953472ca50d682768f1cdf279ac613d15a98d75bfa1ecba86ddea3734613fed: Status 404 returned error can't find the container with id 1953472ca50d682768f1cdf279ac613d15a98d75bfa1ecba86ddea3734613fed Jul 02 23:23:02.425442 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.425412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt"] Jul 02 23:23:02.428266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.428237 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk"] Jul 02 23:23:02.429130 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.429089 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c6f89b_4757_4174_ace6_96c36a9e0706.slice/crio-9dc1a0dbb6882c17659c50e727d675647f69d4176d4d075ff0ede0fa31185d3e WatchSource:0}: Error finding container 9dc1a0dbb6882c17659c50e727d675647f69d4176d4d075ff0ede0fa31185d3e: Status 404 returned error can't find the container with id 9dc1a0dbb6882c17659c50e727d675647f69d4176d4d075ff0ede0fa31185d3e Jul 02 23:23:02.431891 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.431857 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf60fc_1aa9_41c0_bd51_19e1b5b6d5d6.slice/crio-48e6eaffea4405710c3383e164bc3d9d39fc36248a19b0d91e6e74606abc9e15 WatchSource:0}: Error finding container 48e6eaffea4405710c3383e164bc3d9d39fc36248a19b0d91e6e74606abc9e15: Status 404 returned error can't find the container with id 48e6eaffea4405710c3383e164bc3d9d39fc36248a19b0d91e6e74606abc9e15 Jul 02 23:23:02.438463 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.438438 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2g5lx"] Jul 02 23:23:02.440427 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.440407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:02.440525 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.440409 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:02.441116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.440858 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:02.442466 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.442414 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03794c72_91ad_4ff1_87e5_16e6520e708e.slice/crio-f7362f959a55a9e5c8513143892c9df1328e7853ba55903d0c6d43d826d356f2 WatchSource:0}: Error finding container f7362f959a55a9e5c8513143892c9df1328e7853ba55903d0c6d43d826d356f2: Status 404 returned error can't find the container with id f7362f959a55a9e5c8513143892c9df1328e7853ba55903d0c6d43d826d356f2 Jul 02 23:23:02.443213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.442823 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p57lg\"" Jul 02 23:23:02.443213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.442898 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 23:23:02.443213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.443114 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z4l4l\"" Jul 02 23:23:02.443213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.443206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 23:23:02.456820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.456797 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6"] Jul 02 23:23:02.462233 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:02.460493 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9e6769_4db6_494b_a2d5_24f04a403f84.slice/crio-6af887e6c3017af5381cc799fbacd06ab4a0b27586ec653ca30f0e4a711569f7 WatchSource:0}: Error finding container 6af887e6c3017af5381cc799fbacd06ab4a0b27586ec653ca30f0e4a711569f7: Status 404 returned error can't find the container with id 6af887e6c3017af5381cc799fbacd06ab4a0b27586ec653ca30f0e4a711569f7 Jul 02 23:23:02.496251 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.496227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:02.496398 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.496340 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:02.496459 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.496403 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.496385948 +0000 UTC m=+37.578709850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:02.597560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.597469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.597560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.597524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.597567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.597616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597652 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597676 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597654 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.597635784 +0000 UTC m=+37.679959689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597709 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597731 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.597707463 +0000 UTC m=+37.680031372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:02.597750 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597753 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.597743748 +0000 UTC m=+37.680067650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:02.598213 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597770 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.597761886 +0000 UTC m=+37.680085791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:02.598213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.597826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:02.598213 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.597988 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:02.598213 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.598001 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:02.598213 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:02.598039 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:03.598028075 +0000 UTC m=+37.680351980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:02.621336 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.621268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2g5lx" event={"ID":"03794c72-91ad-4ff1-87e5-16e6520e708e","Type":"ContainerStarted","Data":"f7362f959a55a9e5c8513143892c9df1328e7853ba55903d0c6d43d826d356f2"} Jul 02 23:23:02.622453 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.622405 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" event={"ID":"09c6f89b-4757-4174-ace6-96c36a9e0706","Type":"ContainerStarted","Data":"9dc1a0dbb6882c17659c50e727d675647f69d4176d4d075ff0ede0fa31185d3e"} Jul 02 23:23:02.623640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.623616 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" event={"ID":"ab3548bc-f22e-4b6e-a727-a342457af286","Type":"ContainerStarted","Data":"0da97be55dbe02f88c120b034d75af35bf314d90e607c712b052195a196075c0"} Jul 02 23:23:02.624861 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.624812 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" event={"ID":"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d","Type":"ContainerStarted","Data":"1953472ca50d682768f1cdf279ac613d15a98d75bfa1ecba86ddea3734613fed"} Jul 02 23:23:02.626053 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.626031 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" event={"ID":"ae9e6769-4db6-494b-a2d5-24f04a403f84","Type":"ContainerStarted","Data":"6af887e6c3017af5381cc799fbacd06ab4a0b27586ec653ca30f0e4a711569f7"} Jul 02 23:23:02.627253 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.627234 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" event={"ID":"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6","Type":"ContainerStarted","Data":"48e6eaffea4405710c3383e164bc3d9d39fc36248a19b0d91e6e74606abc9e15"} Jul 02 23:23:02.628457 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:02.628435 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" event={"ID":"49a946af-8ab2-4774-a67e-5125b517005e","Type":"ContainerStarted","Data":"f675a45bc16b157b929fe0eb450ae54e2e528951392bd0a4ae8879edfec54300"} Jul 02 23:23:03.507329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.507222 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:03.507769 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.507496 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:03.507769 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.507555 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.50754162 +0000 UTC m=+39.589865522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:03.608508 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.608471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:03.608722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.608527 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:03.608722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.608573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:03.608722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.608619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:03.608722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:03.608668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:03.608979 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.608842 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:03.608979 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.608859 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:03.608979 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.608912 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.608895565 +0000 UTC m=+39.691219470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:03.609353 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609336 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.60931761 +0000 UTC m=+39.691641520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:03.609459 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609420 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:03.609459 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609454 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.609442822 +0000 UTC m=+39.691766725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:03.609542 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609528 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:03.609586 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609556 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.609546163 +0000 UTC m=+39.691870069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:03.609622 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609614 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:03.609654 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:03.609645 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:05.609635141 +0000 UTC m=+39.691959046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:05.390348 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.390306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:05.472645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.472608 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:05.528142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.528104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:05.528392 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.528362 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:05.528458 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.528432 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.528413287 +0000 UTC m=+43.610737193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:05.573337 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.573194 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:05.628816 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.628759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.628864 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.628873 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.628889 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.628899 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.628943 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.628926039 +0000 UTC m=+43.711249962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.628962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:05.628981 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.628984 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:05.629010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629018 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.629007206 +0000 UTC m=+43.711331107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629068 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.62905999 +0000 UTC m=+43.711383896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629120 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629148 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.629139146 +0000 UTC m=+43.711463053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629195 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:05.629450 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:05.629220 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:09.629211212 +0000 UTC m=+43.711535114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:06.020040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:06.019994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:06.118900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:06.118866 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:07.009143 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:07.008902 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:07.107561 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:07.107522 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:07.267730 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:07.267698 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:07.333313 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:07.333251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:09.355107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.355070 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:09.406619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.406586 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:09.570954 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.570916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:09.571147 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.571043 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:09.571147 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.571113 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.571093436 +0000 UTC m=+51.653417338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:09.672241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.672170 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:09.672241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.672210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:09.672241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.672241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672339 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672349 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672359 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672342 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672394 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.672378255 +0000 UTC m=+51.754702158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672410 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.672399055 +0000 UTC m=+51.754722958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672420 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.672415286 +0000 UTC m=+51.754739187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.672443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:09.672463 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:09.672507 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672513 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:09.672852 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672537 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.672531438 +0000 UTC m=+51.754855340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:09.672852 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:09.672586 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:17.672568313 +0000 UTC m=+51.754892219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:11.655583 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.655537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" event={"ID":"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d","Type":"ContainerStarted","Data":"2d968a85b8caf1b7a51a5c9ff49c25cd09229604ec85a1573aca16a9f93f3604"} Jul 02 23:23:11.657922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.657895 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" event={"ID":"ae9e6769-4db6-494b-a2d5-24f04a403f84","Type":"ContainerStarted","Data":"f0b5b3c788703a16174de2c36c5a7f91634edd685f0b36efdbc7161ae62e70e7"} Jul 02 23:23:11.660431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.660408 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" event={"ID":"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6","Type":"ContainerStarted","Data":"cfde9129fc869fd8651343b25c8a2ab7bf06646e88fb06284e5f245fa9ad3b12"} Jul 02 23:23:11.662416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.662395 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" event={"ID":"49a946af-8ab2-4774-a67e-5125b517005e","Type":"ContainerStarted","Data":"50d569c39ba851389593ffff7aea8a933ab1d3d9f4d45951fa33de86d1deedf6"} Jul 02 23:23:11.664547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.664525 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2g5lx" event={"ID":"03794c72-91ad-4ff1-87e5-16e6520e708e","Type":"ContainerStarted","Data":"f8aac79bd76b9e884d29b9d0becb8fdba1553b3d7836d66208de2debd9c390c7"} Jul 02 23:23:11.668241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.668206 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="ee94012affc7612273a303ced9f3c129c171e0fce1455f6866abae8efbc0bcb4" exitCode=0 Jul 02 23:23:11.668331 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.668306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"ee94012affc7612273a303ced9f3c129c171e0fce1455f6866abae8efbc0bcb4"} Jul 02 23:23:11.670422 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.670397 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" event={"ID":"09c6f89b-4757-4174-ace6-96c36a9e0706","Type":"ContainerStarted","Data":"6fbeac51f3738271858b87dc593a2d4f3db5a0c7b9747e828d7c58a5346d7c0e"} Jul 02 23:23:11.672907 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.672885 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/0.log" Jul 02 23:23:11.672992 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.672924 2561 generic.go:356] "Generic (PLEG): container finished" podID="ab3548bc-f22e-4b6e-a727-a342457af286" containerID="a9c640dd6f2f7f6731b0e8c9f3a1c582a8fd430a7cfed10e996ea13d50d9c01c" exitCode=255 Jul 02 23:23:11.672992 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.672950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" event={"ID":"ab3548bc-f22e-4b6e-a727-a342457af286","Type":"ContainerDied","Data":"a9c640dd6f2f7f6731b0e8c9f3a1c582a8fd430a7cfed10e996ea13d50d9c01c"} Jul 02 23:23:11.673220 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.673200 2561 scope.go:117] "RemoveContainer" containerID="a9c640dd6f2f7f6731b0e8c9f3a1c582a8fd430a7cfed10e996ea13d50d9c01c" Jul 02 23:23:11.673444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.673401 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" podStartSLOduration=25.900969398 podStartE2EDuration="34.673387259s" totalImagesPullingTime="8.772417861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.670658857 +0000 UTC m=+45.752982782" watchObservedRunningTime="2026-07-02 23:23:11.673387259 +0000 UTC m=+45.755711183" Jul 02 23:23:11.741202 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.740722 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-x8bpt" podStartSLOduration=30.016308572 podStartE2EDuration="38.740703015s" totalImagesPullingTime="8.724394443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.716838229 +0000 UTC m=+45.799162154" watchObservedRunningTime="2026-07-02 23:23:11.740703015 +0000 UTC m=+45.823026942" Jul 02 23:23:11.741202 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.740818 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" podStartSLOduration=23.018928917 podStartE2EDuration="31.740811014s" totalImagesPullingTime="8.721882097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.739472247 +0000 UTC m=+45.821796172" watchObservedRunningTime="2026-07-02 23:23:11.740811014 +0000 UTC m=+45.823134941" Jul 02 23:23:11.764952 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.764894 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" podStartSLOduration=29.976768651 podStartE2EDuration="38.764873965s" totalImagesPullingTime="8.788105314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.763755589 +0000 UTC m=+45.846079513" watchObservedRunningTime="2026-07-02 23:23:11.764873965 +0000 UTC m=+45.847197890" Jul 02 23:23:11.796343 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.792269 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hw5k6" podStartSLOduration=25.064311367 podStartE2EDuration="33.792248828s" totalImagesPullingTime="8.727937461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.791183682 +0000 UTC m=+45.873507609" watchObservedRunningTime="2026-07-02 23:23:11.792248828 +0000 UTC m=+45.874572753" Jul 02 23:23:11.819339 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.817773 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2g5lx" podStartSLOduration=2.081663095 podStartE2EDuration="10.817751599s" totalImagesPullingTime="8.736088504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:11.816191113 +0000 UTC m=+45.898515038" watchObservedRunningTime="2026-07-02 23:23:11.817751599 +0000 UTC m=+45.900075525" Jul 02 23:23:11.827615 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.827493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:11.879691 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:11.879589 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:12.119300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.119258 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:12.119484 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.119337 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:12.227695 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.227665 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:12.276462 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.276431 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:12.481470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.480099 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-klgwl"] Jul 02 23:23:12.501765 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.501729 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-klgwl"] Jul 02 23:23:12.501899 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.501829 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.504897 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.504300 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 23:23:12.504897 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.504464 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-242f6\"" Jul 02 23:23:12.504897 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.504788 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 23:23:12.518765 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.518743 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:12.571923 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.571882 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:12.601932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.601896 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.601932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.601941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjvm\" (UniqueName: \"kubernetes.io/projected/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-api-access-fsjvm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.602185 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.602139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.602185 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.602166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b18286e6-f2b7-48f6-95bb-e52b53812325-data-volume\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.602278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.602205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b18286e6-f2b7-48f6-95bb-e52b53812325-crio-socket\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.680582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.680553 2561 generic.go:356] "Generic (PLEG): container finished" podID="e24fae4e-4711-4b22-bd1e-066b882107a3" containerID="f395d1e226fbcfc99c8d139a49f7884fae24f6b3095310686486a0e784eb6fc3" exitCode=0 Jul 02 23:23:12.681037 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.680635 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerDied","Data":"f395d1e226fbcfc99c8d139a49f7884fae24f6b3095310686486a0e784eb6fc3"} Jul 02 23:23:12.682754 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.682737 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/1.log" Jul 02 23:23:12.683227 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.683206 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/0.log" Jul 02 23:23:12.683379 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.683245 2561 generic.go:356] "Generic (PLEG): container finished" podID="ab3548bc-f22e-4b6e-a727-a342457af286" containerID="d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82" exitCode=255 Jul 02 23:23:12.683463 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.683396 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" event={"ID":"ab3548bc-f22e-4b6e-a727-a342457af286","Type":"ContainerDied","Data":"d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82"} Jul 02 23:23:12.683521 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.683464 2561 scope.go:117] "RemoveContainer" containerID="a9c640dd6f2f7f6731b0e8c9f3a1c582a8fd430a7cfed10e996ea13d50d9c01c" Jul 02 23:23:12.683579 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.683565 2561 scope.go:117] "RemoveContainer" containerID="d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82" Jul 02 23:23:12.683736 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:12.683716 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-hjwk8_openshift-console-operator(ab3548bc-f22e-4b6e-a727-a342457af286)\"" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" podUID="ab3548bc-f22e-4b6e-a727-a342457af286" Jul 02 23:23:12.702746 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.702721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.702859 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.702763 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b18286e6-f2b7-48f6-95bb-e52b53812325-data-volume\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.702934 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:12.702876 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 23:23:12.703003 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.702925 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b18286e6-f2b7-48f6-95bb-e52b53812325-crio-socket\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.703003 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:12.702947 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:13.202919617 +0000 UTC m=+47.285243523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : secret "insights-runtime-extractor-tls" not found Jul 02 23:23:12.703118 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.703005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.703118 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.703033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjvm\" (UniqueName: \"kubernetes.io/projected/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-api-access-fsjvm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.703118 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.703076 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b18286e6-f2b7-48f6-95bb-e52b53812325-data-volume\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.703257 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:12.703142 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 23:23:12.703257 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.703144 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b18286e6-f2b7-48f6-95bb-e52b53812325-crio-socket\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:12.703257 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:12.703185 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:13.203169793 +0000 UTC m=+47.285493701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : configmap "kube-rbac-proxy" not found Jul 02 23:23:12.725272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:12.715679 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjvm\" (UniqueName: \"kubernetes.io/projected/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-api-access-fsjvm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:13.181079 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.180981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.209142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.209106 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:13.209326 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:13.209252 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 23:23:13.209326 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.209272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:13.209449 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:13.209335 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:14.20931526 +0000 UTC m=+48.291639176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : configmap "kube-rbac-proxy" not found Jul 02 23:23:13.209449 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:13.209411 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 23:23:13.209550 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:13.209461 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:14.209449903 +0000 UTC m=+48.291773805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : secret "insights-runtime-extractor-tls" not found Jul 02 23:23:13.283985 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.283957 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.358376 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.358339 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.414928 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.414900 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.612656 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.612619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.659832 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.659803 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.690306 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.690257 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" event={"ID":"e24fae4e-4711-4b22-bd1e-066b882107a3","Type":"ContainerStarted","Data":"91a057d396994d0c254177271d5cc3028bc29c870d268db8bcec9a59554961f8"} Jul 02 23:23:13.691750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.691733 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/1.log" Jul 02 23:23:13.692007 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.691995 2561 scope.go:117] "RemoveContainer" containerID="d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82" Jul 02 23:23:13.692138 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:13.692122 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-hjwk8_openshift-console-operator(ab3548bc-f22e-4b6e-a727-a342457af286)\"" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" podUID="ab3548bc-f22e-4b6e-a727-a342457af286" Jul 02 23:23:13.710249 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.710210 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kwsbr" podStartSLOduration=8.088284302 podStartE2EDuration="46.710196805s" totalImagesPullingTime="38.621912503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:13.709241263 +0000 UTC m=+47.791565186" watchObservedRunningTime="2026-07-02 23:23:13.710196805 +0000 UTC m=+47.792520726" Jul 02 23:23:13.800093 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.800059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:13.855300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:13.855251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:14.045148 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.045120 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:14.096591 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.096564 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:14.218160 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.218133 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:14.218337 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.218188 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:14.218337 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:14.218277 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 23:23:14.218337 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:14.218300 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 23:23:14.218440 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:14.218359 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:16.218344707 +0000 UTC m=+50.300668612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : configmap "kube-rbac-proxy" not found Jul 02 23:23:14.218440 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:14.218373 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:16.218366867 +0000 UTC m=+50.300690769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : secret "insights-runtime-extractor-tls" not found Jul 02 23:23:14.324076 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.324004 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:14.424470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.424442 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:14.570636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:14.570609 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pq92b_119d9db1-dbe0-4358-a3bb-315aac8af83a/dns-node-resolver/0.log" Jul 02 23:23:15.570559 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:15.570535 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2hx8_e2091d71-7dcc-43ad-807d-3ded4bf54cec/node-ca/0.log" Jul 02 23:23:15.955963 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:15.955893 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:16.009636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.009609 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:16.123861 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.123828 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:16.170881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.170856 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2g5lx_03794c72-91ad-4ff1-87e5-16e6520e708e/serve-healthcheck-canary/0.log" Jul 02 23:23:16.176532 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.176501 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:16.234407 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.234331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:16.234540 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:16.234455 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 23:23:16.234540 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:16.234514 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:20.23450062 +0000 UTC m=+54.316824522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : configmap "kube-rbac-proxy" not found Jul 02 23:23:16.234540 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.234454 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:16.234648 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:16.234522 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 23:23:16.234648 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:16.234632 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:20.234622924 +0000 UTC m=+54.316946826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : secret "insights-runtime-extractor-tls" not found Jul 02 23:23:16.765089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:16.765062 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:17.645340 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.645301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:17.645527 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.645449 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:17.645527 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.645516 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls podName:6162850d-acc9-41e3-83f0-2c324b0e9f94 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.645502076 +0000 UTC m=+67.727825978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-cjjvx" (UID: "6162850d-acc9-41e3-83f0-2c324b0e9f94") : secret "cluster-monitoring-operator-tls" not found Jul 02 23:23:17.745706 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.745675 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:17.745895 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.745726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:17.745895 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745839 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 23:23:17.745895 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745850 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5799577f45-m6qk8: secret "image-registry-tls" not found Jul 02 23:23:17.745895 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.745867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:17.745895 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745889 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls podName:82bf7a3f-a2d2-4726-b0ba-09b323701bce nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.745876847 +0000 UTC m=+67.828200748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls") pod "image-registry-5799577f45-m6qk8" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce") : secret "image-registry-tls" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.745909 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745839 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:17.745939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745956 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.745941646 +0000 UTC m=+67.828265549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : configmap references non-existent config key: service-ca.crt Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745981 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls podName:b3be26b3-caf0-48f7-9ea2-ea70dc6ec703 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.745970711 +0000 UTC m=+67.828294614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hp466" (UID: "b3be26b3-caf0-48f7-9ea2-ea70dc6ec703") : secret "samples-operator-tls" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.745992 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.746030 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls podName:25b4864a-b323-4420-a61c-86f098df9474 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.746018483 +0000 UTC m=+67.828342385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls") pod "dns-default-s5tnc" (UID: "25b4864a-b323-4420-a61c-86f098df9474") : secret "dns-default-metrics-tls" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.746043 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 02 23:23:17.746134 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:17.746075 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs podName:60558dee-31d1-49d1-85ef-6ad6f53e142f nodeName:}" failed. No retries permitted until 2026-07-02 23:23:33.746062977 +0000 UTC m=+67.828386879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs") pod "router-default-5f9cdf7f67-8pwfb" (UID: "60558dee-31d1-49d1-85ef-6ad6f53e142f") : secret "router-metrics-certs-default" not found Jul 02 23:23:18.206196 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:18.206166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:19.217260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:19.217207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:20.266694 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.266662 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:20.267185 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.266776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:20.267185 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:20.266886 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 23:23:20.267185 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:20.266971 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls podName:b18286e6-f2b7-48f6-95bb-e52b53812325 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:28.266956011 +0000 UTC m=+62.349279914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-klgwl" (UID: "b18286e6-f2b7-48f6-95bb-e52b53812325") : secret "insights-runtime-extractor-tls" not found Jul 02 23:23:20.271452 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.267585 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b18286e6-f2b7-48f6-95bb-e52b53812325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:20.467139 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.467082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:20.670137 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.670058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:20.771396 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.771357 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:20.826819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.826791 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:20.997470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:20.997384 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:21.170116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:21.170083 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:22.118470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.118436 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:22.118930 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.118504 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:22.119009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.118929 2561 scope.go:117] "RemoveContainer" containerID="d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82" Jul 02 23:23:22.166856 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.166833 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:22.566086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.566059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:22.720397 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.720370 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:23:22.720788 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.720771 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/1.log" Jul 02 23:23:22.720894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.720810 2561 generic.go:356] "Generic (PLEG): container finished" podID="ab3548bc-f22e-4b6e-a727-a342457af286" containerID="1c18ced76d08aafe78c2809d99f4267272a7bb7fc774bea0f0e4f4685b6cacbe" exitCode=255 Jul 02 23:23:22.720943 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.720886 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" event={"ID":"ab3548bc-f22e-4b6e-a727-a342457af286","Type":"ContainerDied","Data":"1c18ced76d08aafe78c2809d99f4267272a7bb7fc774bea0f0e4f4685b6cacbe"} Jul 02 23:23:22.720943 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.720939 2561 scope.go:117] "RemoveContainer" containerID="d50f600e49c8ac617b72fb25713b47ebf2560350f8e40c9571650eb1353d3f82" Jul 02 23:23:22.721228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.721212 2561 scope.go:117] "RemoveContainer" containerID="1c18ced76d08aafe78c2809d99f4267272a7bb7fc774bea0f0e4f4685b6cacbe" Jul 02 23:23:22.721431 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:22.721410 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-hjwk8_openshift-console-operator(ab3548bc-f22e-4b6e-a727-a342457af286)\"" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" podUID="ab3548bc-f22e-4b6e-a727-a342457af286" Jul 02 23:23:22.771363 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.771335 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:22.967013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:22.966917 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:23.171820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:23.171753 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:23.519876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:23.519837 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:23.725539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:23.725515 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:23:24.171987 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.171946 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.234015 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.233966 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.363109 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.363080 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.462620 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.462542 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.570750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.570715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.766385 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.766355 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:24.970905 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:24.970873 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:25.071276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:25.071186 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:25.171667 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:25.171639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:26.168548 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:26.168512 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:26.568308 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:26.568234 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:26.764338 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:26.764306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:26.816473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:26.816445 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:26.964874 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:26.964792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:27.166812 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:27.166780 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:28.167908 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.167876 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:28.339117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.339084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:28.341430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.341402 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b18286e6-f2b7-48f6-95bb-e52b53812325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klgwl\" (UID: \"b18286e6-f2b7-48f6-95bb-e52b53812325\") " pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:28.417145 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.417114 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-242f6\"" Jul 02 23:23:28.425318 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.425233 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-klgwl" Jul 02 23:23:28.552126 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.550495 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-klgwl"] Jul 02 23:23:28.555817 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:28.555793 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18286e6_f2b7_48f6_95bb_e52b53812325.slice/crio-1a0af2e05d87337b804e5971f79eb4015bd97243fc9c08085d82293f9ff5ee9f WatchSource:0}: Error finding container 1a0af2e05d87337b804e5971f79eb4015bd97243fc9c08085d82293f9ff5ee9f: Status 404 returned error can't find the container with id 1a0af2e05d87337b804e5971f79eb4015bd97243fc9c08085d82293f9ff5ee9f Jul 02 23:23:28.571013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.570991 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:28.620480 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.620454 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qqq9g" Jul 02 23:23:28.740032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.739947 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klgwl" event={"ID":"b18286e6-f2b7-48f6-95bb-e52b53812325","Type":"ContainerStarted","Data":"1a0af2e05d87337b804e5971f79eb4015bd97243fc9c08085d82293f9ff5ee9f"} Jul 02 23:23:28.767224 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.767192 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:28.962507 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:28.962470 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:29.369158 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:29.369095 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:29.969511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:29.969481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:30.564496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:30.564467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:30.768539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:30.768511 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:30.964213 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:30.964125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:31.165634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:31.165603 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:32.076883 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.076842 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:32.077318 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.076898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:32.079750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.079729 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 23:23:32.080846 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.080822 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 23:23:32.090196 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.090175 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3729cf5d-1ad7-4025-bc82-9b93a24d1a44-original-pull-secret\") pod \"global-pull-secret-syncer-rx9mk\" (UID: \"3729cf5d-1ad7-4025-bc82-9b93a24d1a44\") " pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:32.090334 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.090246 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412e9e17-1969-40a1-829b-79346d9d4d34-metrics-certs\") pod \"network-metrics-daemon-59s58\" (UID: \"412e9e17-1969-40a1-829b-79346d9d4d34\") " pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:32.119381 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.119360 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:32.119495 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.119398 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:32.119753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.119737 2561 scope.go:117] "RemoveContainer" containerID="1c18ced76d08aafe78c2809d99f4267272a7bb7fc774bea0f0e4f4685b6cacbe" Jul 02 23:23:32.119971 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:32.119948 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-hjwk8_openshift-console-operator(ab3548bc-f22e-4b6e-a727-a342457af286)\"" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" podUID="ab3548bc-f22e-4b6e-a727-a342457af286" Jul 02 23:23:32.165940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.165907 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z4l4l\"" Jul 02 23:23:32.168846 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.168819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx9mk" Jul 02 23:23:32.173916 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.173895 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59s58" Jul 02 23:23:32.279072 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.278790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:32.283132 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.283107 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqsl\" (UniqueName: \"kubernetes.io/projected/52b04c71-9ea9-4794-91a6-6edb7cbd1d5c-kube-api-access-swqsl\") pod \"network-check-target-bwtfb\" (UID: \"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c\") " pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:32.328379 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.328307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-59s58"] Jul 02 23:23:32.333981 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:32.333246 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412e9e17_1969_40a1_829b_79346d9d4d34.slice/crio-a1cfb2e1ee67ea967f302781be8b771d9e93feb55b4df79cb79ca588d01928f7 WatchSource:0}: Error finding container a1cfb2e1ee67ea967f302781be8b771d9e93feb55b4df79cb79ca588d01928f7: Status 404 returned error can't find the container with id a1cfb2e1ee67ea967f302781be8b771d9e93feb55b4df79cb79ca588d01928f7 Jul 02 23:23:32.346610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.346585 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx9mk"] Jul 02 23:23:32.349328 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:32.349305 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3729cf5d_1ad7_4025_bc82_9b93a24d1a44.slice/crio-6eecd21de3b98c45239377d76f1d7705e1348240424f5da0c15fd29f8a7cb07d WatchSource:0}: Error finding container 6eecd21de3b98c45239377d76f1d7705e1348240424f5da0c15fd29f8a7cb07d: Status 404 returned error can't find the container with id 6eecd21de3b98c45239377d76f1d7705e1348240424f5da0c15fd29f8a7cb07d Jul 02 23:23:32.456990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.456966 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p57lg\"" Jul 02 23:23:32.465127 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.465106 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:32.585439 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.585362 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwtfb"] Jul 02 23:23:32.588449 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:32.588427 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b04c71_9ea9_4794_91a6_6edb7cbd1d5c.slice/crio-3b2363917eeb532c37ed10a1d4f62230abc3b7fbed576036e3fff98eb279e84c WatchSource:0}: Error finding container 3b2363917eeb532c37ed10a1d4f62230abc3b7fbed576036e3fff98eb279e84c: Status 404 returned error can't find the container with id 3b2363917eeb532c37ed10a1d4f62230abc3b7fbed576036e3fff98eb279e84c Jul 02 23:23:32.757965 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.757926 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klgwl" event={"ID":"b18286e6-f2b7-48f6-95bb-e52b53812325","Type":"ContainerStarted","Data":"c9e817c19044a6e64833848691a26b7138e4d7667026480169a8b34f32c3c70b"} Jul 02 23:23:32.759453 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.759424 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwtfb" event={"ID":"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c","Type":"ContainerStarted","Data":"abc7ce740f7b8dbaee9bd87079cfe92c872e05e89b19653bb1159936f6df6408"} Jul 02 23:23:32.759590 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.759508 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwtfb" event={"ID":"52b04c71-9ea9-4794-91a6-6edb7cbd1d5c","Type":"ContainerStarted","Data":"3b2363917eeb532c37ed10a1d4f62230abc3b7fbed576036e3fff98eb279e84c"} Jul 02 23:23:32.759590 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.759538 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:23:32.760662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.760638 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59s58" event={"ID":"412e9e17-1969-40a1-829b-79346d9d4d34","Type":"ContainerStarted","Data":"a1cfb2e1ee67ea967f302781be8b771d9e93feb55b4df79cb79ca588d01928f7"} Jul 02 23:23:32.761750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.761728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx9mk" event={"ID":"3729cf5d-1ad7-4025-bc82-9b93a24d1a44","Type":"ContainerStarted","Data":"6eecd21de3b98c45239377d76f1d7705e1348240424f5da0c15fd29f8a7cb07d"} Jul 02 23:23:32.776075 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:32.776033 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bwtfb" podStartSLOduration=65.776018378 podStartE2EDuration="1m5.776018378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:32.774946251 +0000 UTC m=+66.857270177" watchObservedRunningTime="2026-07-02 23:23:32.776018378 +0000 UTC m=+66.858342302" Jul 02 23:23:33.693064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.692600 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:33.695698 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.695647 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6162850d-acc9-41e3-83f0-2c324b0e9f94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-cjjvx\" (UID: \"6162850d-acc9-41e3-83f0-2c324b0e9f94\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:33.768653 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.768615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klgwl" event={"ID":"b18286e6-f2b7-48f6-95bb-e52b53812325","Type":"ContainerStarted","Data":"ea1e2f18e270ada665fb449de6ba9fbcab4e2aebf640f8f4cf491afe2c56e655"} Jul 02 23:23:33.770217 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.770158 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59s58" event={"ID":"412e9e17-1969-40a1-829b-79346d9d4d34","Type":"ContainerStarted","Data":"0e65f58701988d076a271e2abf2f865473ae72137bb65811dddb7eb90045a1ea"} Jul 02 23:23:33.794042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.793542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:33.794042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.793574 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:33.794042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.793599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:33.794042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.793638 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:33.794042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.793684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:33.794376 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.794239 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60558dee-31d1-49d1-85ef-6ad6f53e142f-service-ca-bundle\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:33.796597 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.796556 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"image-registry-5799577f45-m6qk8\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:33.797405 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.797379 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60558dee-31d1-49d1-85ef-6ad6f53e142f-metrics-certs\") pod \"router-default-5f9cdf7f67-8pwfb\" (UID: \"60558dee-31d1-49d1-85ef-6ad6f53e142f\") " pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:33.800264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.800238 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25b4864a-b323-4420-a61c-86f098df9474-metrics-tls\") pod \"dns-default-s5tnc\" (UID: \"25b4864a-b323-4420-a61c-86f098df9474\") " pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:33.800885 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.800865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3be26b3-caf0-48f7-9ea2-ea70dc6ec703-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hp466\" (UID: \"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:33.893575 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.893494 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7w8v2\"" Jul 02 23:23:33.901423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.901397 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" Jul 02 23:23:33.946153 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.946119 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j45fm\"" Jul 02 23:23:33.954069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.954045 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:33.954296 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.954266 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ztq86\"" Jul 02 23:23:33.960401 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.960248 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d2w2\"" Jul 02 23:23:33.963077 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.962432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:33.968175 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.968125 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" Jul 02 23:23:33.981672 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.981643 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qpvbp\"" Jul 02 23:23:33.990124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:33.990047 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:34.076645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.075840 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx"] Jul 02 23:23:34.164386 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.164279 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:23:34.190419 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:34.190384 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bf7a3f_a2d2_4726_b0ba_09b323701bce.slice/crio-97a2cf4b88f1f96e17ffa98b39c0249cf3b6482c9b8f0fe8c8cae546e03a5c4e WatchSource:0}: Error finding container 97a2cf4b88f1f96e17ffa98b39c0249cf3b6482c9b8f0fe8c8cae546e03a5c4e: Status 404 returned error can't find the container with id 97a2cf4b88f1f96e17ffa98b39c0249cf3b6482c9b8f0fe8c8cae546e03a5c4e Jul 02 23:23:34.203918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.203744 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466"] Jul 02 23:23:34.225710 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.225631 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f9cdf7f67-8pwfb"] Jul 02 23:23:34.234546 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:34.234454 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60558dee_31d1_49d1_85ef_6ad6f53e142f.slice/crio-b6d2ac250cd52eebe394f0a246d149466bd9c9b4867922288ce328d882da4baf WatchSource:0}: Error finding container b6d2ac250cd52eebe394f0a246d149466bd9c9b4867922288ce328d882da4baf: Status 404 returned error can't find the container with id b6d2ac250cd52eebe394f0a246d149466bd9c9b4867922288ce328d882da4baf Jul 02 23:23:34.241699 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.241670 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s5tnc"] Jul 02 23:23:34.252877 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:34.252847 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b4864a_b323_4420_a61c_86f098df9474.slice/crio-6d6662b342b9df2299d7ea0f51a226bd1dcc98a23dad00b18d3ef75df663cf99 WatchSource:0}: Error finding container 6d6662b342b9df2299d7ea0f51a226bd1dcc98a23dad00b18d3ef75df663cf99: Status 404 returned error can't find the container with id 6d6662b342b9df2299d7ea0f51a226bd1dcc98a23dad00b18d3ef75df663cf99 Jul 02 23:23:34.648159 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.648125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:34.748109 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.748031 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:34.778882 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.778844 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" event={"ID":"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703","Type":"ContainerStarted","Data":"7713aafc52a671af5ea05d2f28c925634d509af9e4d97259812cfa7bce2a15a5"} Jul 02 23:23:34.782481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.781613 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59s58" event={"ID":"412e9e17-1969-40a1-829b-79346d9d4d34","Type":"ContainerStarted","Data":"13bdf46b9e1686ff0e5aada41407e9ab832196ffe3305843e2f76a9254c12991"} Jul 02 23:23:34.784252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.784226 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5tnc" event={"ID":"25b4864a-b323-4420-a61c-86f098df9474","Type":"ContainerStarted","Data":"6d6662b342b9df2299d7ea0f51a226bd1dcc98a23dad00b18d3ef75df663cf99"} Jul 02 23:23:34.790278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.790002 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" event={"ID":"60558dee-31d1-49d1-85ef-6ad6f53e142f","Type":"ContainerStarted","Data":"c738fc7b6fc7a4e379c0bc6977d250633c73d5a5c294b9dc65b05999957ec4cf"} Jul 02 23:23:34.790278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.790043 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" event={"ID":"60558dee-31d1-49d1-85ef-6ad6f53e142f","Type":"ContainerStarted","Data":"b6d2ac250cd52eebe394f0a246d149466bd9c9b4867922288ce328d882da4baf"} Jul 02 23:23:34.793822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.793732 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" event={"ID":"82bf7a3f-a2d2-4726-b0ba-09b323701bce","Type":"ContainerStarted","Data":"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058"} Jul 02 23:23:34.793822 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.793774 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" event={"ID":"82bf7a3f-a2d2-4726-b0ba-09b323701bce","Type":"ContainerStarted","Data":"97a2cf4b88f1f96e17ffa98b39c0249cf3b6482c9b8f0fe8c8cae546e03a5c4e"} Jul 02 23:23:34.794328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.794110 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:34.796402 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.796353 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" event={"ID":"6162850d-acc9-41e3-83f0-2c324b0e9f94","Type":"ContainerStarted","Data":"4d59a966ec4455cdb03d4ee6fa1c961ddb1c323d11edd0d2ae4d0de4493f7889"} Jul 02 23:23:34.802647 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.802375 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-59s58" podStartSLOduration=66.52887557 podStartE2EDuration="1m7.802357226s" totalImagesPullingTime="1.273481656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:34.801242513 +0000 UTC m=+68.883566437" watchObservedRunningTime="2026-07-02 23:23:34.802357226 +0000 UTC m=+68.884681151" Jul 02 23:23:34.827180 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.827120 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" podStartSLOduration=67.827098908 podStartE2EDuration="1m7.827098908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:34.825425406 +0000 UTC m=+68.907749330" watchObservedRunningTime="2026-07-02 23:23:34.827098908 +0000 UTC m=+68.909422835" Jul 02 23:23:34.846557 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.846499 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" podStartSLOduration=61.846477221 podStartE2EDuration="1m1.846477221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:34.846312909 +0000 UTC m=+68.928636832" watchObservedRunningTime="2026-07-02 23:23:34.846477221 +0000 UTC m=+68.928801143" Jul 02 23:23:34.908417 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.908253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:34.964752 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.964719 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:34.967831 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.967809 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:34.969433 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:34.969370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:35.570871 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:35.570829 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:35.802657 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:35.802603 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:35.803618 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:35.803593 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f9cdf7f67-8pwfb" Jul 02 23:23:35.963605 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:35.963558 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:36.019819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:36.019783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:36.766677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:36.766641 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:36.820015 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:36.819952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:37.167176 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:37.167080 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:37.260981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:37.260939 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:38.188640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:38.188602 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:38.288838 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:38.288803 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:39.354036 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.354001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:39.403422 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.403388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:39.822473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.822061 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx9mk" event={"ID":"3729cf5d-1ad7-4025-bc82-9b93a24d1a44","Type":"ContainerStarted","Data":"ccce9909ab192100bb6250ea019142f2eb36f83203e282465558f5e4b29924b9"} Jul 02 23:23:39.827154 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.827128 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" event={"ID":"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703","Type":"ContainerStarted","Data":"4047cb9790bf81d1ce4e52da15ea8bc0e8a49b85529d99a10a4e0232ad5bd1c7"} Jul 02 23:23:39.830552 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.829952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klgwl" event={"ID":"b18286e6-f2b7-48f6-95bb-e52b53812325","Type":"ContainerStarted","Data":"97758e4648da19c25ab034aef18a6df0bc2dc451891ba7154de05ab77df12b0e"} Jul 02 23:23:39.832017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.831946 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5tnc" event={"ID":"25b4864a-b323-4420-a61c-86f098df9474","Type":"ContainerStarted","Data":"238a4d66600c45436f6a9a1ba8ff64bea5f6c5eed6674da99a3d88fb672045a3"} Jul 02 23:23:39.836824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.834959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" event={"ID":"6162850d-acc9-41e3-83f0-2c324b0e9f94","Type":"ContainerStarted","Data":"74e97ff3942d59d11c905e0d0ddee58c1d4264b1435fd0bb032ec238b71ddb3f"} Jul 02 23:23:39.838879 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.838635 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rx9mk" podStartSLOduration=65.603002383 podStartE2EDuration="1m12.83861773s" totalImagesPullingTime="7.235615347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:39.838590732 +0000 UTC m=+73.920914658" watchObservedRunningTime="2026-07-02 23:23:39.83861773 +0000 UTC m=+73.920941656" Jul 02 23:23:39.858069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.858021 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-klgwl" podStartSLOduration=16.979662064 podStartE2EDuration="27.857997227s" totalImagesPullingTime="10.878335163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:23:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:39.856912586 +0000 UTC m=+73.939236511" watchObservedRunningTime="2026-07-02 23:23:39.857997227 +0000 UTC m=+73.940321151" Jul 02 23:23:39.874934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:39.874815 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-cjjvx" podStartSLOduration=61.368537751 podStartE2EDuration="1m6.874797819s" totalImagesPullingTime="5.506260068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:39.874474642 +0000 UTC m=+73.956798564" watchObservedRunningTime="2026-07-02 23:23:39.874797819 +0000 UTC m=+73.957121744" Jul 02 23:23:40.068071 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.068037 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:40.250617 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.250526 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:40.323133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.323097 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:40.839783 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.839748 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5tnc" event={"ID":"25b4864a-b323-4420-a61c-86f098df9474","Type":"ContainerStarted","Data":"c3803f5995aa360eb019e93fc3aa2a0764fb4fe2dbbdbd03bb9f0e9883545450"} Jul 02 23:23:40.840199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.839822 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:40.841528 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.841505 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" event={"ID":"b3be26b3-caf0-48f7-9ea2-ea70dc6ec703","Type":"ContainerStarted","Data":"ae0d8b0569aad703079b8a8eaa626e668c2b4a34e45eeebe66efdb5b4f307225"} Jul 02 23:23:40.855092 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.855052 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s5tnc" podStartSLOduration=34.523188276 podStartE2EDuration="39.855037453s" totalImagesPullingTime="5.331849177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:40.853751981 +0000 UTC m=+74.936075915" watchObservedRunningTime="2026-07-02 23:23:40.855037453 +0000 UTC m=+74.937361376" Jul 02 23:23:40.868724 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.868682 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hp466" podStartSLOduration=62.549262701 podStartE2EDuration="1m7.86866912s" totalImagesPullingTime="5.319406419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:40.867486487 +0000 UTC m=+74.949810410" watchObservedRunningTime="2026-07-02 23:23:40.86866912 +0000 UTC m=+74.950993090" Jul 02 23:23:40.921565 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:40.921530 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:43.059380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:43.059339 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:43.110444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:43.110412 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:44.439344 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:44.439310 2561 scope.go:117] "RemoveContainer" containerID="1c18ced76d08aafe78c2809d99f4267272a7bb7fc774bea0f0e4f4685b6cacbe" Jul 02 23:23:44.860439 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:44.860410 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:23:44.860607 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:44.860515 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" event={"ID":"ab3548bc-f22e-4b6e-a727-a342457af286","Type":"ContainerStarted","Data":"2539ac8c0f241eff46eaa6156b80a07756ff7b8b24b78e0292573791131887b5"} Jul 02 23:23:44.860870 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:44.860851 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:44.879188 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:44.879137 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" podStartSLOduration=63.09966371 podStartE2EDuration="1m11.879123878s" totalImagesPullingTime="8.779460168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:44.878215649 +0000 UTC m=+78.960539573" watchObservedRunningTime="2026-07-02 23:23:44.879123878 +0000 UTC m=+78.961447803" Jul 02 23:23:45.017757 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.017722 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-hjwk8" Jul 02 23:23:45.058935 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.058906 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.167633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.167519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.227400 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.227373 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.282451 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.282420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.484084 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.483989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.861854 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.861815 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:45.914306 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:45.914267 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.250628 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.250543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.283235 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.283203 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t"] Jul 02 23:23:46.293490 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.293466 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.294464 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.294434 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t"] Jul 02 23:23:46.295639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.295614 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 23:23:46.295782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.295758 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qccrp\"" Jul 02 23:23:46.296529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.296510 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 23:23:46.296638 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.296516 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 23:23:46.353376 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.353340 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.395598 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.395570 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.395745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.395613 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145dce9-24e4-4c48-a2c7-90d4c55e489b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.395745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.395648 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hdt\" (UniqueName: \"kubernetes.io/projected/8145dce9-24e4-4c48-a2c7-90d4c55e489b-kube-api-access-k2hdt\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.395745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.395667 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.452175 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.452135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.496496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.496466 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.496867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.496516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145dce9-24e4-4c48-a2c7-90d4c55e489b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.496867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.496539 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hdt\" (UniqueName: \"kubernetes.io/projected/8145dce9-24e4-4c48-a2c7-90d4c55e489b-kube-api-access-k2hdt\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.496867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.496558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.497251 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.497233 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145dce9-24e4-4c48-a2c7-90d4c55e489b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.498997 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.498968 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.499112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.499043 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145dce9-24e4-4c48-a2c7-90d4c55e489b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.504867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.504808 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hdt\" (UniqueName: \"kubernetes.io/projected/8145dce9-24e4-4c48-a2c7-90d4c55e489b-kube-api-access-k2hdt\") pod \"prometheus-operator-567f7b8b7c-clm6t\" (UID: \"8145dce9-24e4-4c48-a2c7-90d4c55e489b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.551774 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.551746 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.603481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.603454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" Jul 02 23:23:46.702824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.702579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.747264 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.747235 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t"] Jul 02 23:23:46.750790 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:46.750752 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8145dce9_24e4_4c48_a2c7_90d4c55e489b.slice/crio-6db0a3902b070138ab5b5df82056f2437b2a0e193a32e2d9bb69bd484c8d4c81 WatchSource:0}: Error finding container 6db0a3902b070138ab5b5df82056f2437b2a0e193a32e2d9bb69bd484c8d4c81: Status 404 returned error can't find the container with id 6db0a3902b070138ab5b5df82056f2437b2a0e193a32e2d9bb69bd484c8d4c81 Jul 02 23:23:46.869174 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.869143 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" event={"ID":"8145dce9-24e4-4c48-a2c7-90d4c55e489b","Type":"ContainerStarted","Data":"6db0a3902b070138ab5b5df82056f2437b2a0e193a32e2d9bb69bd484c8d4c81"} Jul 02 23:23:46.871682 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.871658 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:46.970575 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:46.970536 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:47.095547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:47.095516 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:47.193991 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:47.193963 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:47.294177 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:47.294143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:47.395116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:47.395028 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:48.313376 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:48.313346 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:48.878828 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:48.878792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" event={"ID":"8145dce9-24e4-4c48-a2c7-90d4c55e489b","Type":"ContainerStarted","Data":"1500c59625589a11874e27be285ba500f3bff2b512f5c4c13a82e0ff637c3a90"} Jul 02 23:23:48.879001 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:48.878849 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" event={"ID":"8145dce9-24e4-4c48-a2c7-90d4c55e489b","Type":"ContainerStarted","Data":"4149c87bec5a29859c90367a89ab5df191e1efa06f26ebc4b5c616ed95eca861"} Jul 02 23:23:48.894360 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:48.894306 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clm6t" podStartSLOduration=1.580599022 podStartE2EDuration="2.894267573s" totalImagesPullingTime="1.313668551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:48.894044668 +0000 UTC m=+82.976368593" watchObservedRunningTime="2026-07-02 23:23:48.894267573 +0000 UTC m=+82.976591497" Jul 02 23:23:50.401673 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.401645 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:50.593375 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.593336 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:50.653005 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.652932 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l"] Jul 02 23:23:50.656031 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.656014 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.659963 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.659946 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 23:23:50.660463 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.660436 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 23:23:50.660626 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.660613 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p9jwl\"" Jul 02 23:23:50.668600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.668577 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l"] Jul 02 23:23:50.685594 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.685569 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7cb85"] Jul 02 23:23:50.689830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.689808 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:50.692056 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.692040 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.694380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.694361 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 23:23:50.694492 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.694468 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 23:23:50.694554 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.694546 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 23:23:50.694624 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.694606 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sgptq\"" Jul 02 23:23:50.741713 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.741683 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:50.832044 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.832044 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832048 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832081 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-textfile\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832146 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-metrics-client-ca\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832191 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-tls\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832210 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-wtmp\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832271 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-root\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dv2w\" (UniqueName: \"kubernetes.io/projected/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-kube-api-access-5dv2w\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.832372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-sys\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832608 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832391 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288zk\" (UniqueName: \"kubernetes.io/projected/5c3605bf-59df-464d-80eb-2fca1bc62813-kube-api-access-288zk\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.832608 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.832608 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.832450 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.847861 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.847838 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s5tnc" Jul 02 23:23:50.913601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.913524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:50.932772 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932746 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-textfile\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.932890 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-metrics-client-ca\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.932890 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-tls\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.932890 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-wtmp\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933047 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933047 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.932956 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-wtmp\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933144 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-root\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933201 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dv2w\" (UniqueName: \"kubernetes.io/projected/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-kube-api-access-5dv2w\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.933201 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-sys\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288zk\" (UniqueName: \"kubernetes.io/projected/5c3605bf-59df-464d-80eb-2fca1bc62813-kube-api-access-288zk\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933223 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-root\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.933324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933310 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3605bf-59df-464d-80eb-2fca1bc62813-sys\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933527 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.933527 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933368 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.933527 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933670 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933544 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-metrics-client-ca\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.933670 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:50.933580 2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 23:23:50.933670 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:23:50.933635 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls podName:0d0cc391-82c4-4c82-80dc-8dadee4a4a78 nodeName:}" failed. No retries permitted until 2026-07-02 23:23:51.433616367 +0000 UTC m=+85.515940268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-cg79l" (UID: "0d0cc391-82c4-4c82-80dc-8dadee4a4a78") : secret "openshift-state-metrics-tls" not found Jul 02 23:23:50.933824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.933719 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-textfile\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.934121 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.934081 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.934238 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.934160 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.935510 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.935489 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.935510 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.935501 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c3605bf-59df-464d-80eb-2fca1bc62813-node-exporter-tls\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.935858 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.935841 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:50.942151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.942120 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288zk\" (UniqueName: \"kubernetes.io/projected/5c3605bf-59df-464d-80eb-2fca1bc62813-kube-api-access-288zk\") pod \"node-exporter-7cb85\" (UID: \"5c3605bf-59df-464d-80eb-2fca1bc62813\") " pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:50.942252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:50.942179 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dv2w\" (UniqueName: \"kubernetes.io/projected/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-kube-api-access-5dv2w\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:51.001191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.001151 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cb85" Jul 02 23:23:51.012139 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:51.012112 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3605bf_59df_464d_80eb_2fca1bc62813.slice/crio-d5726a2d42d3c7c13982423b8c89f5709cad0df0db54e790b406572d2486652f WatchSource:0}: Error finding container d5726a2d42d3c7c13982423b8c89f5709cad0df0db54e790b406572d2486652f: Status 404 returned error can't find the container with id d5726a2d42d3c7c13982423b8c89f5709cad0df0db54e790b406572d2486652f Jul 02 23:23:51.207766 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.207679 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:51.339369 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.339328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:51.437341 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.437311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:51.439835 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.439811 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0cc391-82c4-4c82-80dc-8dadee4a4a78-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cg79l\" (UID: \"0d0cc391-82c4-4c82-80dc-8dadee4a4a78\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:51.565339 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.565305 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" Jul 02 23:23:51.652979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.652944 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:51.723462 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.723407 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l"] Jul 02 23:23:51.726139 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.726097 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:51.726298 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:23:51.726263 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0cc391_82c4_4c82_80dc_8dadee4a4a78.slice/crio-bd332ea35fb6b0d12c2a819c691ba7e2e4e3ce98febb64b4e4f13505ecb40c74 WatchSource:0}: Error finding container bd332ea35fb6b0d12c2a819c691ba7e2e4e3ce98febb64b4e4f13505ecb40c74: Status 404 returned error can't find the container with id bd332ea35fb6b0d12c2a819c691ba7e2e4e3ce98febb64b4e4f13505ecb40c74 Jul 02 23:23:51.895252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.895152 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" event={"ID":"0d0cc391-82c4-4c82-80dc-8dadee4a4a78","Type":"ContainerStarted","Data":"c72b4a5b684c08aa813c0556f96e2d00c768ff03a4ed3940041b531e4dc5f96c"} Jul 02 23:23:51.895252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.895227 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" event={"ID":"0d0cc391-82c4-4c82-80dc-8dadee4a4a78","Type":"ContainerStarted","Data":"bd332ea35fb6b0d12c2a819c691ba7e2e4e3ce98febb64b4e4f13505ecb40c74"} Jul 02 23:23:51.896735 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:51.896708 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cb85" event={"ID":"5c3605bf-59df-464d-80eb-2fca1bc62813","Type":"ContainerStarted","Data":"d5726a2d42d3c7c13982423b8c89f5709cad0df0db54e790b406572d2486652f"} Jul 02 23:23:52.054214 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.054141 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:52.153915 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.153832 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:52.254207 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.254175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:52.903130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.903090 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" event={"ID":"0d0cc391-82c4-4c82-80dc-8dadee4a4a78","Type":"ContainerStarted","Data":"4b9aceb2f85c1c3ea0ea4860a9b21ecbbaa5285d04079bedeb02bf2b69162e93"} Jul 02 23:23:52.904916 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.904882 2561 generic.go:356] "Generic (PLEG): container finished" podID="5c3605bf-59df-464d-80eb-2fca1bc62813" containerID="d307ab75e59b2f62a6c305b2674339a90262192659e2276878910feaa0677799" exitCode=0 Jul 02 23:23:52.905063 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:52.904970 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cb85" event={"ID":"5c3605bf-59df-464d-80eb-2fca1bc62813","Type":"ContainerDied","Data":"d307ab75e59b2f62a6c305b2674339a90262192659e2276878910feaa0677799"} Jul 02 23:23:53.576995 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.576967 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:53.650850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.650823 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:53.769176 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.769137 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:53.866512 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.866429 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:53.911556 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.911520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" event={"ID":"0d0cc391-82c4-4c82-80dc-8dadee4a4a78","Type":"ContainerStarted","Data":"23896feb191b0008a56f4fd845c6cfdcfb873524cf3cebe00871296d0574a501"} Jul 02 23:23:53.913974 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.913951 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cb85" event={"ID":"5c3605bf-59df-464d-80eb-2fca1bc62813","Type":"ContainerStarted","Data":"91da2904a57766582c04215c123e64fa501df787f6b49f0204d19d544565610c"} Jul 02 23:23:53.914096 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.913992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cb85" event={"ID":"5c3605bf-59df-464d-80eb-2fca1bc62813","Type":"ContainerStarted","Data":"b1f0f925293694ad4362392d513727f2a93dc6b620a1591ab1535a4dca91afc0"} Jul 02 23:23:53.927405 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.927357 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cg79l" podStartSLOduration=2.808987182 podStartE2EDuration="3.927341803s" totalImagesPullingTime="1.118354621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:53.925835819 +0000 UTC m=+88.008159743" watchObservedRunningTime="2026-07-02 23:23:53.927341803 +0000 UTC m=+88.009665727" Jul 02 23:23:53.941807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:53.941766 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7cb85" podStartSLOduration=2.850849897 podStartE2EDuration="3.941751205s" totalImagesPullingTime="1.009907358s" totalInitContainerRuntime="80.99395ms" isStatefulPod=true podCreationTimestamp="2026-07-02 23:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:53.941036345 +0000 UTC m=+88.023360269" watchObservedRunningTime="2026-07-02 23:23:53.941751205 +0000 UTC m=+88.024075129" Jul 02 23:23:55.414475 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:55.414447 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:55.514514 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:55.514486 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:55.726035 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:55.725952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:55.807414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:55.807385 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:23:55.966725 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:55.966693 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:56.016561 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:56.016522 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:56.831248 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:56.831189 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:57.167156 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:57.167059 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:23:57.234346 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:57.234301 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:57.308445 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:57.308412 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:57.965806 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:57.965777 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:58.065075 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:58.065034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:59.962195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:23:59.962165 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:00.059745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:00.059712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:00.886539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:00.886503 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:00.938199 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:00.938152 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:03.773191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:03.773163 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bwtfb" Jul 02 23:24:04.688324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.688275 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:24:04.837972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.837937 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:24:04.838397 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.838030 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.840919 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.840881 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 23:24:04.841050 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.840930 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 23:24:04.841112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.841071 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 23:24:04.841972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.841942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 23:24:04.842086 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.842015 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fqzxt\"" Jul 02 23:24:04.842191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.842169 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 23:24:04.842353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.842220 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 23:24:04.842353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.842256 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 23:24:04.845948 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.845930 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 23:24:04.947524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947453 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947504 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947534 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsp7n\" (UniqueName: \"kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947641 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947773 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:04.947773 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:04.947716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049145 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsp7n\" (UniqueName: \"kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049266 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.049933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049902 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.050052 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.049957 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.050183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.050161 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.050245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.050184 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.052040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.052014 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.052135 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.052022 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.059108 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.059086 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsp7n\" (UniqueName: \"kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n\") pod \"console-747cc8994-p8f68\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.114195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.114159 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:05.148075 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.148048 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:05.214722 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.214648 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:05.302124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.302039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:24:05.304766 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:24:05.304732 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda89365_272d_40c3_bf50_73b5159380bb.slice/crio-069571f3be762e5ea7d643fe16e0029c10fb4dc373f78c7601a77bb1ac7e7883 WatchSource:0}: Error finding container 069571f3be762e5ea7d643fe16e0029c10fb4dc373f78c7601a77bb1ac7e7883: Status 404 returned error can't find the container with id 069571f3be762e5ea7d643fe16e0029c10fb4dc373f78c7601a77bb1ac7e7883 Jul 02 23:24:05.887444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.887406 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:05.939173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.939140 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:05.961480 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:05.961444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747cc8994-p8f68" event={"ID":"cda89365-272d-40c3-bf50-73b5159380bb","Type":"ContainerStarted","Data":"069571f3be762e5ea7d643fe16e0029c10fb4dc373f78c7601a77bb1ac7e7883"} Jul 02 23:24:08.971820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:08.971780 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:08.975640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:08.975612 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747cc8994-p8f68" event={"ID":"cda89365-272d-40c3-bf50-73b5159380bb","Type":"ContainerStarted","Data":"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12"} Jul 02 23:24:08.992091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:08.992050 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747cc8994-p8f68" podStartSLOduration=2.367377638 podStartE2EDuration="4.992036516s" totalImagesPullingTime="2.624658878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:24:08.990796844 +0000 UTC m=+103.073120769" watchObservedRunningTime="2026-07-02 23:24:08.992036516 +0000 UTC m=+103.074360439" Jul 02 23:24:09.072214 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:09.072186 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:09.360567 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:09.360534 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:09.462382 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:09.462353 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:10.295018 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:10.294986 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:15.148476 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:15.148439 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:15.148852 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:15.148496 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:15.153090 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:15.153069 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:16.005443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:16.005415 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:24:17.488788 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:17.488752 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:17.539198 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:17.539169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:18.010872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:18.010841 2561 generic.go:356] "Generic (PLEG): container finished" podID="8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6" containerID="cfde9129fc869fd8651343b25c8a2ab7bf06646e88fb06284e5f245fa9ad3b12" exitCode=0 Jul 02 23:24:18.011031 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:18.010916 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" event={"ID":"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6","Type":"ContainerDied","Data":"cfde9129fc869fd8651343b25c8a2ab7bf06646e88fb06284e5f245fa9ad3b12"} Jul 02 23:24:18.011252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:18.011239 2561 scope.go:117] "RemoveContainer" containerID="cfde9129fc869fd8651343b25c8a2ab7bf06646e88fb06284e5f245fa9ad3b12" Jul 02 23:24:19.016501 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:19.016466 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6vzdk" event={"ID":"8cbf60fc-1aa9-41c0-bd51-19e1b5b6d5d6","Type":"ContainerStarted","Data":"0e3a4d4d75e3f133cc8478937157074c6cc37caea587de204b6c9622625e1f1e"} Jul 02 23:24:19.690072 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:19.690041 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:19.739898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:19.739856 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:22.191013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.190952 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" podUID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" containerName="registry" containerID="cri-o://95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058" gracePeriod=30 Jul 02 23:24:22.426721 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.426699 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:24:22.501506 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501426 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501506 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501463 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501506 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501486 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501739 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501528 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501739 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501545 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501739 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501593 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501739 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501619 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.501739 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.501650 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lv7s\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s\") pod \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\" (UID: \"82bf7a3f-a2d2-4726-b0ba-09b323701bce\") " Jul 02 23:24:22.502104 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.502070 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:24:22.502259 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.502163 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:24:22.504308 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.504264 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s" (OuterVolumeSpecName: "kube-api-access-9lv7s") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "kube-api-access-9lv7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:24:22.504308 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.504272 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:24:22.504477 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.504375 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:24:22.504477 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.504425 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:24:22.504687 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.504667 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:24:22.510603 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.510582 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "82bf7a3f-a2d2-4726-b0ba-09b323701bce" (UID: "82bf7a3f-a2d2-4726-b0ba-09b323701bce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:24:22.603228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603192 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-bound-sa-token\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603227 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-image-registry-private-configuration\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603243 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82bf7a3f-a2d2-4726-b0ba-09b323701bce-installation-pull-secrets\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603254 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-certificates\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603264 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82bf7a3f-a2d2-4726-b0ba-09b323701bce-trusted-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603273 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82bf7a3f-a2d2-4726-b0ba-09b323701bce-ca-trust-extracted\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603305 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-registry-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:22.603449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:22.603318 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lv7s\" (UniqueName: \"kubernetes.io/projected/82bf7a3f-a2d2-4726-b0ba-09b323701bce-kube-api-access-9lv7s\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:24:23.033047 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.033011 2561 generic.go:356] "Generic (PLEG): container finished" podID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" containerID="95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058" exitCode=0 Jul 02 23:24:23.033218 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.033060 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" Jul 02 23:24:23.033218 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.033085 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" event={"ID":"82bf7a3f-a2d2-4726-b0ba-09b323701bce","Type":"ContainerDied","Data":"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058"} Jul 02 23:24:23.033218 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.033149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5799577f45-m6qk8" event={"ID":"82bf7a3f-a2d2-4726-b0ba-09b323701bce","Type":"ContainerDied","Data":"97a2cf4b88f1f96e17ffa98b39c0249cf3b6482c9b8f0fe8c8cae546e03a5c4e"} Jul 02 23:24:23.033218 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.033182 2561 scope.go:117] "RemoveContainer" containerID="95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058" Jul 02 23:24:23.035027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.035006 2561 generic.go:356] "Generic (PLEG): container finished" podID="aa35770d-8b0e-42b8-bed0-b4f90a9eed3d" containerID="2d968a85b8caf1b7a51a5c9ff49c25cd09229604ec85a1573aca16a9f93f3604" exitCode=0 Jul 02 23:24:23.035134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.035076 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" event={"ID":"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d","Type":"ContainerDied","Data":"2d968a85b8caf1b7a51a5c9ff49c25cd09229604ec85a1573aca16a9f93f3604"} Jul 02 23:24:23.035529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.035513 2561 scope.go:117] "RemoveContainer" containerID="2d968a85b8caf1b7a51a5c9ff49c25cd09229604ec85a1573aca16a9f93f3604" Jul 02 23:24:23.046091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.046072 2561 scope.go:117] "RemoveContainer" containerID="95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058" Jul 02 23:24:23.046387 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:24:23.046361 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058\": container with ID starting with 95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058 not found: ID does not exist" containerID="95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058" Jul 02 23:24:23.046462 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.046404 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058"} err="failed to get container status \"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058\": rpc error: code = NotFound desc = could not find container \"95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058\": container with ID starting with 95d8c78c96c20423754f1e4de0c760a686d89f644956b75073581fca01daa058 not found: ID does not exist" Jul 02 23:24:23.068068 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.068039 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:24:23.071760 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:23.071738 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5799577f45-m6qk8"] Jul 02 23:24:24.046981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:24.046950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-vtf6x" event={"ID":"aa35770d-8b0e-42b8-bed0-b4f90a9eed3d","Type":"ContainerStarted","Data":"f9c0eb0244cc795e95997870afc0c39d23df77c63ef54c0c25518ed5323146ea"} Jul 02 23:24:24.348532 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:24.348450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:24.444431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:24.444394 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" path="/var/lib/kubelet/pods/82bf7a3f-a2d2-4726-b0ba-09b323701bce/volumes" Jul 02 23:24:24.448969 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:24.448945 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:42.674841 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:42.674810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:42.774527 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:42.774489 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:47.132344 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:47.132307 2561 generic.go:356] "Generic (PLEG): container finished" podID="49a946af-8ab2-4774-a67e-5125b517005e" containerID="50d569c39ba851389593ffff7aea8a933ab1d3d9f4d45951fa33de86d1deedf6" exitCode=0 Jul 02 23:24:47.132727 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:47.132311 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" event={"ID":"49a946af-8ab2-4774-a67e-5125b517005e","Type":"ContainerDied","Data":"50d569c39ba851389593ffff7aea8a933ab1d3d9f4d45951fa33de86d1deedf6"} Jul 02 23:24:47.132727 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:47.132720 2561 scope.go:117] "RemoveContainer" containerID="50d569c39ba851389593ffff7aea8a933ab1d3d9f4d45951fa33de86d1deedf6" Jul 02 23:24:47.701168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:47.701137 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:48.137970 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:24:48.137937 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wjjdb" event={"ID":"49a946af-8ab2-4774-a67e-5125b517005e","Type":"ContainerStarted","Data":"c04e451344c3b8dc849b0e8545070c3afa96ae120eeb61fe0c921c09f2220dfc"} Jul 02 23:25:09.812793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:09.812756 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:10.068919 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:10.068816 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:10.277781 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:10.277747 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:18.247954 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.247819 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:25:18.249140 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.249013 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" containerName="registry" Jul 02 23:25:18.249306 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.249273 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" containerName="registry" Jul 02 23:25:18.250940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.250915 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bf7a3f-a2d2-4726-b0ba-09b323701bce" containerName="registry" Jul 02 23:25:18.254100 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.254076 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.264410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.264385 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:25:18.343544 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.343507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:18.345509 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345480 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ct7g\" (UniqueName: \"kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345552 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345673 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.345850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.345695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.444709 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.444680 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:18.446786 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446760 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.446950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.446950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446818 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.446950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446840 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.446950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ct7g\" (UniqueName: \"kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447209 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446966 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447209 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.446987 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447657 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.447633 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.447684 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447856 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.447836 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.447912 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.447891 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.449909 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.449887 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.450044 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.450023 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.455690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.455668 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ct7g\" (UniqueName: \"kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g\") pod \"console-5f6dd4f9c4-mj7lw\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.564890 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.564846 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:18.694972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:18.694924 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:25:18.697532 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:25:18.697505 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c87490d_ecd6_4157_9045_dc17c65c3cfd.slice/crio-394b96b858a27a3239ecc98039410ea73701d94beeca8fb0e947ec0f07f873fe WatchSource:0}: Error finding container 394b96b858a27a3239ecc98039410ea73701d94beeca8fb0e947ec0f07f873fe: Status 404 returned error can't find the container with id 394b96b858a27a3239ecc98039410ea73701d94beeca8fb0e947ec0f07f873fe Jul 02 23:25:19.257329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:19.257276 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6dd4f9c4-mj7lw" event={"ID":"8c87490d-ecd6-4157-9045-dc17c65c3cfd","Type":"ContainerStarted","Data":"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd"} Jul 02 23:25:19.257797 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:19.257339 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6dd4f9c4-mj7lw" event={"ID":"8c87490d-ecd6-4157-9045-dc17c65c3cfd","Type":"ContainerStarted","Data":"394b96b858a27a3239ecc98039410ea73701d94beeca8fb0e947ec0f07f873fe"} Jul 02 23:25:19.274837 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:19.274786 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f6dd4f9c4-mj7lw" podStartSLOduration=1.274772574 podStartE2EDuration="1.274772574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:25:19.272648107 +0000 UTC m=+173.354972033" watchObservedRunningTime="2026-07-02 23:25:19.274772574 +0000 UTC m=+173.357096497" Jul 02 23:25:28.565845 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:28.565810 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:28.566245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:28.565866 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:28.570360 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:28.570337 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:28.812465 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:28.812433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:29.299920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:29.299896 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:25:29.345091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:29.345060 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:25:29.445311 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:29.445265 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:29.490337 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:29.490303 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:29.548862 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:29.548833 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:40.821122 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:40.821087 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:54.371240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.371194 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-747cc8994-p8f68" podUID="cda89365-272d-40c3-bf50-73b5159380bb" containerName="console" containerID="cri-o://8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12" gracePeriod=15 Jul 02 23:25:54.612698 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.612676 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:25:54.628871 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.628806 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsp7n\" (UniqueName: \"kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.628871 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.628841 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.628882 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.628919 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.628946 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629026 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629060 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle\") pod \"cda89365-272d-40c3-bf50-73b5159380bb\" (UID: \"cda89365-272d-40c3-bf50-73b5159380bb\") " Jul 02 23:25:54.629872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629450 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:25:54.629872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629523 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config" (OuterVolumeSpecName: "console-config") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:25:54.629872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629738 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:25:54.629872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.629837 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:25:54.632089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.632058 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:25:54.632089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.632074 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:25:54.632252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.632196 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n" (OuterVolumeSpecName: "kube-api-access-qsp7n") pod "cda89365-272d-40c3-bf50-73b5159380bb" (UID: "cda89365-272d-40c3-bf50-73b5159380bb"). InnerVolumeSpecName "kube-api-access-qsp7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:25:54.730414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730381 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-trusted-ca-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730411 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsp7n\" (UniqueName: \"kubernetes.io/projected/cda89365-272d-40c3-bf50-73b5159380bb-kube-api-access-qsp7n\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730421 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-oauth-serving-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730431 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-serving-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730440 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cda89365-272d-40c3-bf50-73b5159380bb-console-oauth-config\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730449 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-console-config\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:54.730632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:54.730471 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cda89365-272d-40c3-bf50-73b5159380bb-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:25:55.393341 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.393303 2561 generic.go:356] "Generic (PLEG): container finished" podID="cda89365-272d-40c3-bf50-73b5159380bb" containerID="8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12" exitCode=0 Jul 02 23:25:55.393690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.393349 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747cc8994-p8f68" Jul 02 23:25:55.393690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.393383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747cc8994-p8f68" event={"ID":"cda89365-272d-40c3-bf50-73b5159380bb","Type":"ContainerDied","Data":"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12"} Jul 02 23:25:55.393690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.393443 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747cc8994-p8f68" event={"ID":"cda89365-272d-40c3-bf50-73b5159380bb","Type":"ContainerDied","Data":"069571f3be762e5ea7d643fe16e0029c10fb4dc373f78c7601a77bb1ac7e7883"} Jul 02 23:25:55.393690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.393465 2561 scope.go:117] "RemoveContainer" containerID="8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12" Jul 02 23:25:55.402732 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.402715 2561 scope.go:117] "RemoveContainer" containerID="8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12" Jul 02 23:25:55.402986 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:25:55.402969 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12\": container with ID starting with 8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12 not found: ID does not exist" containerID="8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12" Jul 02 23:25:55.403034 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.402993 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12"} err="failed to get container status \"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12\": rpc error: code = NotFound desc = could not find container \"8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12\": container with ID starting with 8c5966a2e15efb22db096b243a99adf58f7f8fafe8b472358f7ae53e407e7d12 not found: ID does not exist" Jul 02 23:25:55.412718 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.412695 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:25:55.416717 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:55.416696 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-747cc8994-p8f68"] Jul 02 23:25:56.443123 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:25:56.443091 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda89365-272d-40c3-bf50-73b5159380bb" path="/var/lib/kubelet/pods/cda89365-272d-40c3-bf50-73b5159380bb/volumes" Jul 02 23:26:17.971263 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:17.971227 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:18.019552 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:18.019519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:25.018636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:25.018602 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:25.070361 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:25.070325 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:29.661653 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:29.661622 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:29.711651 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:29.711607 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:37.016215 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:37.016171 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:37.065144 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:37.065113 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:40.978833 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:40.978790 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:26:41.073274 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:41.073237 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:41.121795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:41.121741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:41.175788 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:41.175751 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:41.226340 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:41.226307 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:54.366999 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:54.366903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:26:54.465817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:26:54.465754 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:06.001509 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.001465 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f6dd4f9c4-mj7lw" podUID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" containerName="console" containerID="cri-o://dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd" gracePeriod=15 Jul 02 23:27:06.248047 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.248019 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:27:06.281738 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281712 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.281848 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281769 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.281848 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281787 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.281918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281893 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.281955 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281931 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.281997 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.281972 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.282044 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.282002 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ct7g\" (UniqueName: \"kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g\") pod \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\" (UID: \"8c87490d-ecd6-4157-9045-dc17c65c3cfd\") " Jul 02 23:27:06.282127 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.282102 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca" (OuterVolumeSpecName: "service-ca") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:27:06.282246 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.282111 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:27:06.282385 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.282311 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config" (OuterVolumeSpecName: "console-config") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:27:06.282385 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.282358 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:27:06.284115 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.284082 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:27:06.284216 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.284152 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:27:06.284216 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.284165 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g" (OuterVolumeSpecName: "kube-api-access-7ct7g") pod "8c87490d-ecd6-4157-9045-dc17c65c3cfd" (UID: "8c87490d-ecd6-4157-9045-dc17c65c3cfd"). InnerVolumeSpecName "kube-api-access-7ct7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:27:06.382593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382557 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-oauth-config\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382586 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382595 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-config\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382603 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c87490d-ecd6-4157-9045-dc17c65c3cfd-console-serving-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382840 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382612 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-oauth-serving-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382840 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382620 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ct7g\" (UniqueName: \"kubernetes.io/projected/8c87490d-ecd6-4157-9045-dc17c65c3cfd-kube-api-access-7ct7g\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.382840 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.382628 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c87490d-ecd6-4157-9045-dc17c65c3cfd-trusted-ca-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:06.648599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.648519 2561 generic.go:356] "Generic (PLEG): container finished" podID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" containerID="dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd" exitCode=0 Jul 02 23:27:06.648599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.648559 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6dd4f9c4-mj7lw" Jul 02 23:27:06.648772 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.648610 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6dd4f9c4-mj7lw" event={"ID":"8c87490d-ecd6-4157-9045-dc17c65c3cfd","Type":"ContainerDied","Data":"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd"} Jul 02 23:27:06.648772 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.648660 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6dd4f9c4-mj7lw" event={"ID":"8c87490d-ecd6-4157-9045-dc17c65c3cfd","Type":"ContainerDied","Data":"394b96b858a27a3239ecc98039410ea73701d94beeca8fb0e947ec0f07f873fe"} Jul 02 23:27:06.648772 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.648681 2561 scope.go:117] "RemoveContainer" containerID="dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd" Jul 02 23:27:06.656655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.656634 2561 scope.go:117] "RemoveContainer" containerID="dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd" Jul 02 23:27:06.656930 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:27:06.656911 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd\": container with ID starting with dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd not found: ID does not exist" containerID="dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd" Jul 02 23:27:06.657001 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.656937 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd"} err="failed to get container status \"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd\": rpc error: code = NotFound desc = could not find container \"dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd\": container with ID starting with dc48889f26b6cb2d09662bcfbc8746f54ce65e39e915fc739fb3a57e23f88abd not found: ID does not exist" Jul 02 23:27:06.664830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.664805 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:27:06.668026 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:06.668007 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6dd4f9c4-mj7lw"] Jul 02 23:27:08.443506 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:08.443473 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" path="/var/lib/kubelet/pods/8c87490d-ecd6-4157-9045-dc17c65c3cfd/volumes" Jul 02 23:27:15.288779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.288748 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:15.384650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.384617 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:15.830330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830299 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx"] Jul 02 23:27:15.830573 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830562 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda89365-272d-40c3-bf50-73b5159380bb" containerName="console" Jul 02 23:27:15.830640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830575 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda89365-272d-40c3-bf50-73b5159380bb" containerName="console" Jul 02 23:27:15.830640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830591 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" containerName="console" Jul 02 23:27:15.830640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830596 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" containerName="console" Jul 02 23:27:15.830749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830645 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c87490d-ecd6-4157-9045-dc17c65c3cfd" containerName="console" Jul 02 23:27:15.830749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.830653 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda89365-272d-40c3-bf50-73b5159380bb" containerName="console" Jul 02 23:27:15.834470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.834454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.836852 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.836828 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcxqr\"" Jul 02 23:27:15.836973 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.836858 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 23:27:15.837756 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.837742 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 23:27:15.839476 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.839454 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.839593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.839573 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.839720 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.839697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhqn\" (UniqueName: \"kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.843061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.843031 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx"] Jul 02 23:27:15.940744 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.940703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhqn\" (UniqueName: \"kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.940744 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.940745 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.940935 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.940783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.941097 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.941082 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.941133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.941118 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:15.948876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:15.948852 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhqn\" (UniqueName: \"kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:16.148495 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:16.148389 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:16.273840 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:16.273685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx"] Jul 02 23:27:16.276459 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:27:16.276432 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad73e4b3_d6d4_44bb_9779_f03e885f6de9.slice/crio-816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea WatchSource:0}: Error finding container 816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea: Status 404 returned error can't find the container with id 816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea Jul 02 23:27:16.688478 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:16.688443 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" event={"ID":"ad73e4b3-d6d4-44bb-9779-f03e885f6de9","Type":"ContainerStarted","Data":"816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea"} Jul 02 23:27:22.714619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:22.714588 2561 generic.go:356] "Generic (PLEG): container finished" podID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerID="bebe860ed48876bc6a6b0d9f78a2da5db11a3bbf0d3d8f71d68181a14a4c9c50" exitCode=0 Jul 02 23:27:22.715010 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:22.714675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" event={"ID":"ad73e4b3-d6d4-44bb-9779-f03e885f6de9","Type":"ContainerDied","Data":"bebe860ed48876bc6a6b0d9f78a2da5db11a3bbf0d3d8f71d68181a14a4c9c50"} Jul 02 23:27:25.730228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:25.730196 2561 generic.go:356] "Generic (PLEG): container finished" podID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerID="99e93c642645a3de72806f0fffaf5aa2dc5b012d8b2512a079ddd6ccb49e44aa" exitCode=0 Jul 02 23:27:25.730639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:25.730265 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" event={"ID":"ad73e4b3-d6d4-44bb-9779-f03e885f6de9","Type":"ContainerDied","Data":"99e93c642645a3de72806f0fffaf5aa2dc5b012d8b2512a079ddd6ccb49e44aa"} Jul 02 23:27:26.299542 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:26.299514 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:27:26.303867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:26.303840 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:27:26.309961 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:26.309928 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 23:27:32.761469 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:32.761384 2561 generic.go:356] "Generic (PLEG): container finished" podID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerID="6c4f28b5d712aef490b674298251e9c536731f1c67f0bfca895cb5915e91b96c" exitCode=0 Jul 02 23:27:32.761842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:32.761466 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" event={"ID":"ad73e4b3-d6d4-44bb-9779-f03e885f6de9","Type":"ContainerDied","Data":"6c4f28b5d712aef490b674298251e9c536731f1c67f0bfca895cb5915e91b96c"} Jul 02 23:27:33.887499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.887476 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:33.988793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.988745 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle\") pod \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " Jul 02 23:27:33.988793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.988794 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnhqn\" (UniqueName: \"kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn\") pod \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " Jul 02 23:27:33.988995 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.988823 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util\") pod \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\" (UID: \"ad73e4b3-d6d4-44bb-9779-f03e885f6de9\") " Jul 02 23:27:33.989496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.989419 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle" (OuterVolumeSpecName: "bundle") pod "ad73e4b3-d6d4-44bb-9779-f03e885f6de9" (UID: "ad73e4b3-d6d4-44bb-9779-f03e885f6de9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:27:33.991112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.991075 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn" (OuterVolumeSpecName: "kube-api-access-bnhqn") pod "ad73e4b3-d6d4-44bb-9779-f03e885f6de9" (UID: "ad73e4b3-d6d4-44bb-9779-f03e885f6de9"). InnerVolumeSpecName "kube-api-access-bnhqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:27:33.993342 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:33.993314 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util" (OuterVolumeSpecName: "util") pod "ad73e4b3-d6d4-44bb-9779-f03e885f6de9" (UID: "ad73e4b3-d6d4-44bb-9779-f03e885f6de9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:27:34.089737 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.089659 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:34.089737 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.089685 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:34.089737 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.089696 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhqn\" (UniqueName: \"kubernetes.io/projected/ad73e4b3-d6d4-44bb-9779-f03e885f6de9-kube-api-access-bnhqn\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:34.770794 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.770756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" event={"ID":"ad73e4b3-d6d4-44bb-9779-f03e885f6de9","Type":"ContainerDied","Data":"816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea"} Jul 02 23:27:34.770794 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.770789 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dz9bx" Jul 02 23:27:34.771018 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:34.770815 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816ff60b6dca087b6a1bd3c2de9a990e191cab698879d516e25fd2770e1e5fea" Jul 02 23:27:36.795458 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:36.795425 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:38.061780 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.061742 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:38.552696 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552664 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl"] Jul 02 23:27:38.552945 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552932 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="util" Jul 02 23:27:38.552990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552947 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="util" Jul 02 23:27:38.552990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552959 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="pull" Jul 02 23:27:38.552990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552965 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="pull" Jul 02 23:27:38.552990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552971 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="extract" Jul 02 23:27:38.552990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.552977 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="extract" Jul 02 23:27:38.553135 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.553018 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad73e4b3-d6d4-44bb-9779-f03e885f6de9" containerName="extract" Jul 02 23:27:38.576203 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.576169 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl"] Jul 02 23:27:38.576353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.576265 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.578881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.578852 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:27:38.579042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.579022 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 02 23:27:38.579116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.579068 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2zpgj\"" Jul 02 23:27:38.624898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.624861 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be947b1-d421-41ee-92e7-a14cd15403e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.625055 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.624930 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hq9\" (UniqueName: \"kubernetes.io/projected/1be947b1-d421-41ee-92e7-a14cd15403e8-kube-api-access-z9hq9\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.725994 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.725964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hq9\" (UniqueName: \"kubernetes.io/projected/1be947b1-d421-41ee-92e7-a14cd15403e8-kube-api-access-z9hq9\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.726118 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.726010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be947b1-d421-41ee-92e7-a14cd15403e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.726397 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.726382 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be947b1-d421-41ee-92e7-a14cd15403e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.733988 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.733958 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hq9\" (UniqueName: \"kubernetes.io/projected/1be947b1-d421-41ee-92e7-a14cd15403e8-kube-api-access-z9hq9\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlmxl\" (UID: \"1be947b1-d421-41ee-92e7-a14cd15403e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:38.885210 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:38.885132 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" Jul 02 23:27:39.036443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:39.036420 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl"] Jul 02 23:27:39.039467 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:27:39.039427 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be947b1_d421_41ee_92e7_a14cd15403e8.slice/crio-2af3fa043e64051bfced0c2c66d4cedb8ce57669ba337e19589cb07cdfa2b46d WatchSource:0}: Error finding container 2af3fa043e64051bfced0c2c66d4cedb8ce57669ba337e19589cb07cdfa2b46d: Status 404 returned error can't find the container with id 2af3fa043e64051bfced0c2c66d4cedb8ce57669ba337e19589cb07cdfa2b46d Jul 02 23:27:39.042911 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:39.042894 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:27:39.791136 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:39.791098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" event={"ID":"1be947b1-d421-41ee-92e7-a14cd15403e8","Type":"ContainerStarted","Data":"2af3fa043e64051bfced0c2c66d4cedb8ce57669ba337e19589cb07cdfa2b46d"} Jul 02 23:27:40.184314 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:40.184205 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:40.241168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:40.241126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:44.785173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:44.785104 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:44.834921 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:44.834889 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:45.818990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:45.818960 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" event={"ID":"1be947b1-d421-41ee-92e7-a14cd15403e8","Type":"ContainerStarted","Data":"9af62c1206384e4e154d546bb72e11c7ef544e47b5240c0d8c6fd12f266ee0da"} Jul 02 23:27:45.844369 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:45.844322 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlmxl" podStartSLOduration=1.5907194059999998 podStartE2EDuration="7.844305081s" totalImagesPullingTime="6.253585675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:27:45.842232513 +0000 UTC m=+319.924556438" watchObservedRunningTime="2026-07-02 23:27:45.844305081 +0000 UTC m=+319.926629001" Jul 02 23:27:46.024726 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:46.024692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:46.155346 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:46.155232 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:46.266416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:46.266382 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:46.388298 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:46.388264 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:47.195790 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.195753 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqhmf"] Jul 02 23:27:47.198194 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.198173 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.200828 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.200808 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dhmbt\"" Jul 02 23:27:47.200963 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.200892 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 23:27:47.201755 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.201736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 23:27:47.212096 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.212072 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqhmf"] Jul 02 23:27:47.292837 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.292799 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtrd\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-kube-api-access-ngtrd\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.293004 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.292872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.394176 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.394121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.394176 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.394188 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtrd\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-kube-api-access-ngtrd\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.402907 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.402857 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.403032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.402940 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtrd\" (UniqueName: \"kubernetes.io/projected/df33ff2f-38e6-4f54-a434-c0fbf5c908b9-kube-api-access-ngtrd\") pod \"cert-manager-webhook-6486ffd997-cqhmf\" (UID: \"df33ff2f-38e6-4f54-a434-c0fbf5c908b9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.523279 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.523250 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:47.604949 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.604401 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg"] Jul 02 23:27:47.607948 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.607863 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.610639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.610328 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcxqr\"" Jul 02 23:27:47.610639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.610328 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 23:27:47.610639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.610516 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 23:27:47.615456 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.615434 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg"] Jul 02 23:27:47.654859 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.654837 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqhmf"] Jul 02 23:27:47.657147 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:27:47.657125 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf33ff2f_38e6_4f54_a434_c0fbf5c908b9.slice/crio-4d9ef0725c267ed1417132eb017e07d583a14b0bff6212e8526362ca22889f3b WatchSource:0}: Error finding container 4d9ef0725c267ed1417132eb017e07d583a14b0bff6212e8526362ca22889f3b: Status 404 returned error can't find the container with id 4d9ef0725c267ed1417132eb017e07d583a14b0bff6212e8526362ca22889f3b Jul 02 23:27:47.696782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.696747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.696782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.696782 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.696999 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.696816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwr5t\" (UniqueName: \"kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.798224 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.798124 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.798224 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.798171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.798224 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.798209 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr5t\" (UniqueName: \"kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.798530 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.798501 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.798589 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.798547 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.805943 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.805919 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr5t\" (UniqueName: \"kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:47.828267 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.828242 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" event={"ID":"df33ff2f-38e6-4f54-a434-c0fbf5c908b9","Type":"ContainerStarted","Data":"4d9ef0725c267ed1417132eb017e07d583a14b0bff6212e8526362ca22889f3b"} Jul 02 23:27:47.921369 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:47.921330 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:48.048886 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.048856 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg"] Jul 02 23:27:48.051133 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:27:48.051105 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5992f9_1f9d_4fe0_91ec_ef472dcf4e6c.slice/crio-7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880 WatchSource:0}: Error finding container 7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880: Status 404 returned error can't find the container with id 7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880 Jul 02 23:27:48.408795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.408712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:48.628842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.628814 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:48.833695 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.833661 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerID="1419636b9c1b8d36b30a93dc77e23deb31a1b2ae3c102bc9fa88fbec873e36ea" exitCode=0 Jul 02 23:27:48.833897 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.833750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" event={"ID":"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c","Type":"ContainerDied","Data":"1419636b9c1b8d36b30a93dc77e23deb31a1b2ae3c102bc9fa88fbec873e36ea"} Jul 02 23:27:48.833897 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:48.833841 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" event={"ID":"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c","Type":"ContainerStarted","Data":"7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880"} Jul 02 23:27:52.854659 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:52.854626 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerID="b814c395cc8c213fb7fee38aa6ef25c38d6856a7ea28404b7e0a2bd19b054053" exitCode=0 Jul 02 23:27:52.855081 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:52.854708 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" event={"ID":"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c","Type":"ContainerDied","Data":"b814c395cc8c213fb7fee38aa6ef25c38d6856a7ea28404b7e0a2bd19b054053"} Jul 02 23:27:52.856327 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:52.856302 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" event={"ID":"df33ff2f-38e6-4f54-a434-c0fbf5c908b9","Type":"ContainerStarted","Data":"aa60316082877ffc0de25234cbde850ddfe01eea56393a656aeaca43a04a6346"} Jul 02 23:27:52.856450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:52.856406 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:52.888814 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:52.888773 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" podStartSLOduration=1.077301576 podStartE2EDuration="5.888759211s" totalImagesPullingTime="4.811457635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:27:52.888145911 +0000 UTC m=+326.970469824" watchObservedRunningTime="2026-07-02 23:27:52.888759211 +0000 UTC m=+326.971083135" Jul 02 23:27:53.862355 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:53.862315 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerID="f6adfe3dde3a331ca3205e1e9be31cc07f931b6fe37f5c606937eccaa55f3d5c" exitCode=0 Jul 02 23:27:53.862729 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:53.862403 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" event={"ID":"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c","Type":"ContainerDied","Data":"f6adfe3dde3a331ca3205e1e9be31cc07f931b6fe37f5c606937eccaa55f3d5c"} Jul 02 23:27:54.990465 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:54.990440 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:55.059361 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.059323 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util\") pod \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " Jul 02 23:27:55.059525 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.059383 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle\") pod \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " Jul 02 23:27:55.059525 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.059432 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwr5t\" (UniqueName: \"kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t\") pod \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\" (UID: \"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c\") " Jul 02 23:27:55.059695 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.059673 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle" (OuterVolumeSpecName: "bundle") pod "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" (UID: "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:27:55.061619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.061587 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t" (OuterVolumeSpecName: "kube-api-access-fwr5t") pod "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" (UID: "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c"). InnerVolumeSpecName "kube-api-access-fwr5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:27:55.063537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.063515 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util" (OuterVolumeSpecName: "util") pod "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" (UID: "7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:27:55.160188 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.160108 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:55.160188 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.160134 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:55.160188 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.160145 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwr5t\" (UniqueName: \"kubernetes.io/projected/7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c-kube-api-access-fwr5t\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:27:55.872654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.872623 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" Jul 02 23:27:55.872838 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.872622 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5ddsg" event={"ID":"7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c","Type":"ContainerDied","Data":"7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880"} Jul 02 23:27:55.872838 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:55.872743 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acec497e380609aa0de1c18e97602b291ee307bd9955a6406d7b7348d0bb880" Jul 02 23:27:56.032004 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.031973 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:56.497068 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497032 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmz5l"] Jul 02 23:27:56.497334 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497321 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="extract" Jul 02 23:27:56.497389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497335 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="extract" Jul 02 23:27:56.497389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497350 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="pull" Jul 02 23:27:56.497389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497355 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="pull" Jul 02 23:27:56.497389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497364 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="util" Jul 02 23:27:56.497389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497369 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="util" Jul 02 23:27:56.497551 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.497424 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5992f9-1f9d-4fe0-91ec-ef472dcf4e6c" containerName="extract" Jul 02 23:27:56.499212 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.499197 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.501442 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.501422 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qrw8n\"" Jul 02 23:27:56.507891 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.507869 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmz5l"] Jul 02 23:27:56.572600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.572559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.572600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.572603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x85g\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-kube-api-access-8x85g\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.673188 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.673144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x85g\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-kube-api-access-8x85g\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.673410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.673246 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.680900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.680864 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.681016 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.680984 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x85g\" (UniqueName: \"kubernetes.io/projected/9ff32d67-7964-4d4f-b5f3-c7d191d69fdc-kube-api-access-8x85g\") pod \"cert-manager-55b4d954d5-bmz5l\" (UID: \"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc\") " pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.808357 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.808324 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bmz5l" Jul 02 23:27:56.937428 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:56.937405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmz5l"] Jul 02 23:27:56.939868 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:27:56.939838 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ff32d67_7964_4d4f_b5f3_c7d191d69fdc.slice/crio-efa47fdf683f3a98886d0bb23f5ac212784552ac0187cb509850e349b929982a WatchSource:0}: Error finding container efa47fdf683f3a98886d0bb23f5ac212784552ac0187cb509850e349b929982a: Status 404 returned error can't find the container with id efa47fdf683f3a98886d0bb23f5ac212784552ac0187cb509850e349b929982a Jul 02 23:27:57.882139 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:57.882099 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bmz5l" event={"ID":"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc","Type":"ContainerStarted","Data":"af2b722519fcc77a56f52abd96d184456db96acb7a0374ad0ca3a85a610c39ee"} Jul 02 23:27:57.882551 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:57.882147 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bmz5l" event={"ID":"9ff32d67-7964-4d4f-b5f3-c7d191d69fdc","Type":"ContainerStarted","Data":"efa47fdf683f3a98886d0bb23f5ac212784552ac0187cb509850e349b929982a"} Jul 02 23:27:57.896758 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:57.896709 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bmz5l" podStartSLOduration=1.896693647 podStartE2EDuration="1.896693647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:27:57.895696314 +0000 UTC m=+331.978020239" watchObservedRunningTime="2026-07-02 23:27:57.896693647 +0000 UTC m=+331.979017571" Jul 02 23:27:58.865749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:58.865719 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cqhmf" Jul 02 23:27:59.078262 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:59.078224 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:59.177862 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:27:59.177778 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:11.027342 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.027304 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:11.124793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.124759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:11.807397 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.807361 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85"] Jul 02 23:28:11.809807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.809785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:11.812277 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.812255 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 23:28:11.812388 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.812257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcxqr\"" Jul 02 23:28:11.813061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.813040 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 23:28:11.818901 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.818874 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85"] Jul 02 23:28:11.996423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.996388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:11.996423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.996424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:11.996633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:11.996446 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2hs\" (UniqueName: \"kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.097854 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.097770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.097854 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.097809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.097854 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.097829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2hs\" (UniqueName: \"kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.098254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.098216 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.098254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.098229 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.106205 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.106176 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2hs\" (UniqueName: \"kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.119935 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.119909 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:12.247653 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.247596 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85"] Jul 02 23:28:12.249799 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:12.249773 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb41fb1_b1f0_4d76_910e_7e1a2b5810a6.slice/crio-a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb WatchSource:0}: Error finding container a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb: Status 404 returned error can't find the container with id a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb Jul 02 23:28:12.945974 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.945940 2561 generic.go:356] "Generic (PLEG): container finished" podID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerID="d249af09fb163b358fd89f96740ed0d4c6f5831e73eef38a00416a5de7901341" exitCode=0 Jul 02 23:28:12.946151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.945984 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" event={"ID":"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6","Type":"ContainerDied","Data":"d249af09fb163b358fd89f96740ed0d4c6f5831e73eef38a00416a5de7901341"} Jul 02 23:28:12.946151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.946022 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" event={"ID":"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6","Type":"ContainerStarted","Data":"a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb"} Jul 02 23:28:12.948654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:12.948629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:13.952313 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:13.952257 2561 generic.go:356] "Generic (PLEG): container finished" podID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerID="6b5bfeda738f07355a3af1f06fdf397f379345a4e2d9e386d11014ad134f8112" exitCode=0 Jul 02 23:28:13.952736 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:13.952345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" event={"ID":"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6","Type":"ContainerDied","Data":"6b5bfeda738f07355a3af1f06fdf397f379345a4e2d9e386d11014ad134f8112"} Jul 02 23:28:14.959051 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:14.959020 2561 generic.go:356] "Generic (PLEG): container finished" podID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerID="b325b7775d212a5ba7a626fab7de0b21629a3baf9ae3a62c4837dc6ce815c32d" exitCode=0 Jul 02 23:28:14.959445 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:14.959079 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" event={"ID":"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6","Type":"ContainerDied","Data":"b325b7775d212a5ba7a626fab7de0b21629a3baf9ae3a62c4837dc6ce815c32d"} Jul 02 23:28:16.087428 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.087402 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:16.230903 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.230810 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util\") pod \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " Jul 02 23:28:16.230903 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.230851 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle\") pod \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " Jul 02 23:28:16.230903 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.230888 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh2hs\" (UniqueName: \"kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs\") pod \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\" (UID: \"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6\") " Jul 02 23:28:16.231687 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.231651 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle" (OuterVolumeSpecName: "bundle") pod "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" (UID: "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:28:16.233255 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.233235 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs" (OuterVolumeSpecName: "kube-api-access-lh2hs") pod "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" (UID: "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6"). InnerVolumeSpecName "kube-api-access-lh2hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:28:16.236537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.236513 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util" (OuterVolumeSpecName: "util") pod "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" (UID: "beb41fb1-b1f0-4d76-910e-7e1a2b5810a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:28:16.331641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.331600 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:16.331641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.331633 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:16.331641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.331646 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2hs\" (UniqueName: \"kubernetes.io/projected/beb41fb1-b1f0-4d76-910e-7e1a2b5810a6-kube-api-access-lh2hs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:16.969187 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.969157 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" Jul 02 23:28:16.969187 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.969162 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg8b85" event={"ID":"beb41fb1-b1f0-4d76-910e-7e1a2b5810a6","Type":"ContainerDied","Data":"a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb"} Jul 02 23:28:16.969466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:16.969206 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87ad03fb7df54bb3e39d25086a5c52099a890971c8ab18983c6fd218a452bcb" Jul 02 23:28:24.877394 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:24.877316 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:24.979221 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:24.979194 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:27.541452 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:27.541419 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:27.641321 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:27.641275 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.159081 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.159048 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.406527 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.406490 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.506444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.506370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.524714 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.524686 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj"] Jul 02 23:28:28.524985 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.524970 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="util" Jul 02 23:28:28.525028 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.524988 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="util" Jul 02 23:28:28.525028 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.525004 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="extract" Jul 02 23:28:28.525028 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.525010 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="extract" Jul 02 23:28:28.525028 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.525022 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="pull" Jul 02 23:28:28.525166 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.525033 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="pull" Jul 02 23:28:28.525166 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.525088 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb41fb1-b1f0-4d76-910e-7e1a2b5810a6" containerName="extract" Jul 02 23:28:28.529372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.529355 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.531980 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.531961 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 23:28:28.532107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.532013 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 23:28:28.532107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.532068 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 23:28:28.532418 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.532403 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 23:28:28.532465 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.532417 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h86qq\"" Jul 02 23:28:28.542913 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.542895 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj"] Jul 02 23:28:28.616863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.616836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-tmp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.616863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.616865 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.617094 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.616884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.617094 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.617008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-manifests\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.617094 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.617055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpzp\" (UniqueName: \"kubernetes.io/projected/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-kube-api-access-hvpzp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.650975 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.650943 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.718126 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718094 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-tmp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.718317 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.718317 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718154 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.718317 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718211 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-manifests\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.718317 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpzp\" (UniqueName: \"kubernetes.io/projected/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-kube-api-access-hvpzp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.718651 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.718612 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-manifests\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.720547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.720525 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-tmp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.720662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.720645 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.720748 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.720730 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.726396 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.726369 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpzp\" (UniqueName: \"kubernetes.io/projected/73066ccd-4d2a-4aea-8dae-8a1d88ffff8e-kube-api-access-hvpzp\") pod \"opendatahub-operator-controller-manager-6f45646c76-9vpxj\" (UID: \"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.751933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.751909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.839271 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.839237 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:28.853157 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.853129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.953469 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.953441 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:28.971773 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:28.971750 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj"] Jul 02 23:28:28.974396 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:28.974370 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73066ccd_4d2a_4aea_8dae_8a1d88ffff8e.slice/crio-c3bfa7534d6ec2fc87081d1d3dc1796f8fcd6ddd8b106f08aba3bff8c5257542 WatchSource:0}: Error finding container c3bfa7534d6ec2fc87081d1d3dc1796f8fcd6ddd8b106f08aba3bff8c5257542: Status 404 returned error can't find the container with id c3bfa7534d6ec2fc87081d1d3dc1796f8fcd6ddd8b106f08aba3bff8c5257542 Jul 02 23:28:29.030629 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.030599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" event={"ID":"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e","Type":"ContainerStarted","Data":"c3bfa7534d6ec2fc87081d1d3dc1796f8fcd6ddd8b106f08aba3bff8c5257542"} Jul 02 23:28:29.033071 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.033049 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:29.125805 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.125728 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:29.228829 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.228794 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:29.326676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.326632 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:29.443454 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:29.443359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:32.047593 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:32.047558 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" event={"ID":"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e","Type":"ContainerStarted","Data":"8bd8c2f0e31d73b998e50fd630e5a29190cb2db7c213bf27f613793c5dd5b8f8"} Jul 02 23:28:33.054337 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:33.054276 2561 generic.go:356] "Generic (PLEG): container finished" podID="73066ccd-4d2a-4aea-8dae-8a1d88ffff8e" containerID="8bd8c2f0e31d73b998e50fd630e5a29190cb2db7c213bf27f613793c5dd5b8f8" exitCode=0 Jul 02 23:28:33.054757 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:33.054388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" event={"ID":"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e","Type":"ContainerDied","Data":"8bd8c2f0e31d73b998e50fd630e5a29190cb2db7c213bf27f613793c5dd5b8f8"} Jul 02 23:28:34.061013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:34.060974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" event={"ID":"73066ccd-4d2a-4aea-8dae-8a1d88ffff8e","Type":"ContainerStarted","Data":"eea6708cbe5cb6d2ced4cafbd2a68c2c6be48fe0d8c446ab64d9e5cda6caefe1"} Jul 02 23:28:34.061502 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:34.061052 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:34.084130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:34.084073 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" podStartSLOduration=1.724793338 podStartE2EDuration="6.08405833s" totalImagesPullingTime="3.294154333s" totalInitContainerRuntime="1.065110659s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:34.0815798 +0000 UTC m=+368.163903725" watchObservedRunningTime="2026-07-02 23:28:34.08405833 +0000 UTC m=+368.166382254" Jul 02 23:28:45.069507 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:45.069443 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f45646c76-9vpxj" Jul 02 23:28:47.486577 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:47.486543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:47.594842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:47.594811 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:47.909640 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:47.909608 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.003858 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.003823 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-576b444b4-hm9tw"] Jul 02 23:28:48.007249 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.007230 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.009806 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.009781 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 23:28:48.009922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.009813 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-48t4x\"" Jul 02 23:28:48.009922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.009782 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 23:28:48.018134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.018109 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b444b4-hm9tw"] Jul 02 23:28:48.062271 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.062238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tmp\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.062437 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.062295 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vnq\" (UniqueName: \"kubernetes.io/projected/ec117db5-3707-4617-b5e7-feea5fcc3ddd-kube-api-access-67vnq\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.062437 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.062351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tls-certs\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.085420 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.085394 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.163707 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.163629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vnq\" (UniqueName: \"kubernetes.io/projected/ec117db5-3707-4617-b5e7-feea5fcc3ddd-kube-api-access-67vnq\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.163707 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.163669 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tls-certs\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.163910 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.163734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tmp\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.166114 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.166092 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tmp\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.166352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.166336 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec117db5-3707-4617-b5e7-feea5fcc3ddd-tls-certs\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.173210 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.173183 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqbw7"] Jul 02 23:28:48.177082 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.177064 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.179644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.179621 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 02 23:28:48.179750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.179682 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bl8wr\"" Jul 02 23:28:48.182911 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.182888 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vnq\" (UniqueName: \"kubernetes.io/projected/ec117db5-3707-4617-b5e7-feea5fcc3ddd-kube-api-access-67vnq\") pod \"kube-auth-proxy-576b444b4-hm9tw\" (UID: \"ec117db5-3707-4617-b5e7-feea5fcc3ddd\") " pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.189324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.189251 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqbw7"] Jul 02 23:28:48.212622 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.212592 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.264540 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.264507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.264699 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.264571 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86w6f\" (UniqueName: \"kubernetes.io/projected/0853faf4-ef79-430b-b4b8-fb0ddfa95883-kube-api-access-86w6f\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.316329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.316300 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" Jul 02 23:28:48.316706 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.316684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.365641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.365607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.365830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.365686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86w6f\" (UniqueName: \"kubernetes.io/projected/0853faf4-ef79-430b-b4b8-fb0ddfa95883-kube-api-access-86w6f\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.365830 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:28:48.365759 2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 02 23:28:48.366001 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:28:48.365837 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert podName:0853faf4-ef79-430b-b4b8-fb0ddfa95883 nodeName:}" failed. No retries permitted until 2026-07-02 23:28:48.865821622 +0000 UTC m=+382.948145529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert") pod "odh-model-controller-7998c67dc4-hqbw7" (UID: "0853faf4-ef79-430b-b4b8-fb0ddfa95883") : secret "odh-model-controller-webhook-cert" not found Jul 02 23:28:48.384062 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.384029 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86w6f\" (UniqueName: \"kubernetes.io/projected/0853faf4-ef79-430b-b4b8-fb0ddfa95883-kube-api-access-86w6f\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.494942 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:48.494915 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec117db5_3707_4617_b5e7_feea5fcc3ddd.slice/crio-88bd3bad61000a8101ee902f59a48044a592dc0ecb9cda4b4fac48cc9acd37e1 WatchSource:0}: Error finding container 88bd3bad61000a8101ee902f59a48044a592dc0ecb9cda4b4fac48cc9acd37e1: Status 404 returned error can't find the container with id 88bd3bad61000a8101ee902f59a48044a592dc0ecb9cda4b4fac48cc9acd37e1 Jul 02 23:28:48.508165 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.508140 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b444b4-hm9tw"] Jul 02 23:28:48.567537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.567507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.698520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.698451 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.747393 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.747313 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-szhhz"] Jul 02 23:28:48.751984 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.751965 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.754423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.754405 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 02 23:28:48.761752 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.761733 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-szhhz"] Jul 02 23:28:48.868106 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.868074 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:48.869202 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.869181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.869261 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.869245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606f8fa2-c606-43cf-a864-93775055fc32-tls-certs\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.869320 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.869302 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvf44\" (UniqueName: \"kubernetes.io/projected/606f8fa2-c606-43cf-a864-93775055fc32-kube-api-access-hvf44\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.871817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.871800 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0853faf4-ef79-430b-b4b8-fb0ddfa95883-cert\") pod \"odh-model-controller-7998c67dc4-hqbw7\" (UID: \"0853faf4-ef79-430b-b4b8-fb0ddfa95883\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:48.970537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.970507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf44\" (UniqueName: \"kubernetes.io/projected/606f8fa2-c606-43cf-a864-93775055fc32-kube-api-access-hvf44\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.970682 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.970578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606f8fa2-c606-43cf-a864-93775055fc32-tls-certs\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.973016 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.972988 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/606f8fa2-c606-43cf-a864-93775055fc32-tls-certs\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:48.978438 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:48.978421 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvf44\" (UniqueName: \"kubernetes.io/projected/606f8fa2-c606-43cf-a864-93775055fc32-kube-api-access-hvf44\") pod \"model-serving-api-54d89bd5fc-szhhz\" (UID: \"606f8fa2-c606-43cf-a864-93775055fc32\") " pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:49.062254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.062221 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:49.094777 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.094712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:49.096666 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.096640 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:49.106193 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.106154 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:49.109828 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.109807 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.113050 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.112700 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 23:28:49.113050 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.112926 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 23:28:49.113230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.113106 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 23:28:49.113345 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.113326 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 23:28:49.113833 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.113676 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q7q9m\"" Jul 02 23:28:49.127873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.127849 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:49.141415 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.141356 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" event={"ID":"ec117db5-3707-4617-b5e7-feea5fcc3ddd","Type":"ContainerStarted","Data":"88bd3bad61000a8101ee902f59a48044a592dc0ecb9cda4b4fac48cc9acd37e1"} Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7sj\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.173524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.172852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.200711 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.200594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:49.228522 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.228307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-szhhz"] Jul 02 23:28:49.233187 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:49.233146 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606f8fa2_c606_43cf_a864_93775055fc32.slice/crio-780effb6f092a5ba01f28c0931f0d077f17bdb55da19b0cfd83e67dbb6ae6eb4 WatchSource:0}: Error finding container 780effb6f092a5ba01f28c0931f0d077f17bdb55da19b0cfd83e67dbb6ae6eb4: Status 404 returned error can't find the container with id 780effb6f092a5ba01f28c0931f0d077f17bdb55da19b0cfd83e67dbb6ae6eb4 Jul 02 23:28:49.274168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274357 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274180 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274357 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274254 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274357 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274357 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274569 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.274569 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7sj\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.275033 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.274976 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.276933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.276889 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.277145 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.277128 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.277366 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.277347 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.277849 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.277831 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.280901 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.280879 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqbw7"] Jul 02 23:28:49.284140 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:49.284113 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0853faf4_ef79_430b_b4b8_fb0ddfa95883.slice/crio-5b6b58e08cd14270689358764e632ab50d6a4978c0cc4937d423233e1306e57e WatchSource:0}: Error finding container 5b6b58e08cd14270689358764e632ab50d6a4978c0cc4937d423233e1306e57e: Status 404 returned error can't find the container with id 5b6b58e08cd14270689358764e632ab50d6a4978c0cc4937d423233e1306e57e Jul 02 23:28:49.284262 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.284240 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7sj\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.285412 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.285372 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token\") pod \"istiod-openshift-gateway-7574459656-kpszf\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.426616 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.426524 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:49.586031 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.585959 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:49.591228 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:49.591194 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a84fe1f_8e83_4bd8_a34c_64ca6a1fcd51.slice/crio-9b83c21b261897f54c5009906364d0a609f1e4258ce37f79c6fc3db9d387dee6 WatchSource:0}: Error finding container 9b83c21b261897f54c5009906364d0a609f1e4258ce37f79c6fc3db9d387dee6: Status 404 returned error can't find the container with id 9b83c21b261897f54c5009906364d0a609f1e4258ce37f79c6fc3db9d387dee6 Jul 02 23:28:49.966855 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:49.966818 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:50.150180 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:50.150141 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" event={"ID":"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51","Type":"ContainerStarted","Data":"9b83c21b261897f54c5009906364d0a609f1e4258ce37f79c6fc3db9d387dee6"} Jul 02 23:28:50.151892 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:50.151859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" event={"ID":"0853faf4-ef79-430b-b4b8-fb0ddfa95883","Type":"ContainerStarted","Data":"5b6b58e08cd14270689358764e632ab50d6a4978c0cc4937d423233e1306e57e"} Jul 02 23:28:50.153832 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:50.153802 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" event={"ID":"606f8fa2-c606-43cf-a864-93775055fc32","Type":"ContainerStarted","Data":"780effb6f092a5ba01f28c0931f0d077f17bdb55da19b0cfd83e67dbb6ae6eb4"} Jul 02 23:28:54.383781 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:54.383745 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:28:54.384079 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:54.383807 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:28:55.181138 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.181097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" event={"ID":"0853faf4-ef79-430b-b4b8-fb0ddfa95883","Type":"ContainerStarted","Data":"ec0e2ed23cd02ddeb63271d07e7a4a9b14842cd9c50c86a3b879a9006cd36a4a"} Jul 02 23:28:55.181410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.181217 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:28:55.182764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.182738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" event={"ID":"606f8fa2-c606-43cf-a864-93775055fc32","Type":"ContainerStarted","Data":"2844a4ba8addd5308d7757cc4f9ec05cd966840ee5a37655a1ff258a8f7503ed"} Jul 02 23:28:55.182898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.182868 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:28:55.184350 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.184320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" event={"ID":"ec117db5-3707-4617-b5e7-feea5fcc3ddd","Type":"ContainerStarted","Data":"9169189bebc90569705b310c61a9f367e6b25b1cc5e9cb9e3741fb8569111911"} Jul 02 23:28:55.186235 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.186212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" event={"ID":"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51","Type":"ContainerStarted","Data":"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056"} Jul 02 23:28:55.186522 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.186489 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:55.188821 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.188789 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kpszf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 23:28:55.188934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.188860 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 23:28:55.196449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.196411 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" podStartSLOduration=2.084554023 podStartE2EDuration="7.196398007s" totalImagesPullingTime="5.111843984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:55.195130164 +0000 UTC m=+389.277454104" watchObservedRunningTime="2026-07-02 23:28:55.196398007 +0000 UTC m=+389.278721930" Jul 02 23:28:55.212369 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.212323 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" podStartSLOduration=1.422781735 podStartE2EDuration="6.212307552s" totalImagesPullingTime="4.789525817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:55.211587453 +0000 UTC m=+389.293911378" watchObservedRunningTime="2026-07-02 23:28:55.212307552 +0000 UTC m=+389.294631467" Jul 02 23:28:55.225194 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.225152 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-576b444b4-hm9tw" podStartSLOduration=2.344912926 podStartE2EDuration="8.225136778s" totalImagesPullingTime="5.880223852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:55.224403231 +0000 UTC m=+389.306727155" watchObservedRunningTime="2026-07-02 23:28:55.225136778 +0000 UTC m=+389.307460706" Jul 02 23:28:55.239654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.239600 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" podStartSLOduration=2.094364988 podStartE2EDuration="7.239581327s" totalImagesPullingTime="5.145216339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:55.238410336 +0000 UTC m=+389.320734274" watchObservedRunningTime="2026-07-02 23:28:55.239581327 +0000 UTC m=+389.321905252" Jul 02 23:28:55.625817 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.625778 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:55.729754 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:55.729721 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:56.191751 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.191720 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kpszf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 23:28:56.192013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.191810 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 23:28:56.350110 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.350045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:56.483372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.483184 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:56.576598 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.576566 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:56.696645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.696611 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct"] Jul 02 23:28:56.700295 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.700267 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.729230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.729204 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct"] Jul 02 23:28:56.849312 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849255 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849312 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5m7v\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-kube-api-access-h5m7v\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849381 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849411 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.849537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.849511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950383 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950383 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950399 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950622 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950520 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950622 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950576 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950622 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5m7v\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-kube-api-access-h5m7v\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950628 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.950771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.950643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.951214 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.951126 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.952942 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.952914 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.953062 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.953013 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.953441 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.953413 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.953534 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.953472 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.958023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.957995 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:56.958731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:56.958714 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5m7v\" (UniqueName: \"kubernetes.io/projected/c8c37a13-c7e3-426b-8c57-1f83d5d70ec3-kube-api-access-h5m7v\") pod \"istiod-openshift-gateway-94bb8fbfd-9jfct\" (UID: \"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:57.009034 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:57.009000 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:57.178154 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:57.178130 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct"] Jul 02 23:28:57.180076 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:28:57.180040 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c37a13_c7e3_426b_8c57_1f83d5d70ec3.slice/crio-0dc8a3b7bd1fb7e9d5b45710c2cc7a755e87d29b9fec77d3b20bb87cd77d8ecd WatchSource:0}: Error finding container 0dc8a3b7bd1fb7e9d5b45710c2cc7a755e87d29b9fec77d3b20bb87cd77d8ecd: Status 404 returned error can't find the container with id 0dc8a3b7bd1fb7e9d5b45710c2cc7a755e87d29b9fec77d3b20bb87cd77d8ecd Jul 02 23:28:57.182159 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:57.182130 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:28:57.182248 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:57.182184 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:28:57.199242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:57.199209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" event={"ID":"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3","Type":"ContainerStarted","Data":"0dc8a3b7bd1fb7e9d5b45710c2cc7a755e87d29b9fec77d3b20bb87cd77d8ecd"} Jul 02 23:28:58.205747 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.205713 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" event={"ID":"c8c37a13-c7e3-426b-8c57-1f83d5d70ec3","Type":"ContainerStarted","Data":"3182c0d94811353c7192117c55788e11c53e14d4f05350df3f12328050eeae1c"} Jul 02 23:28:58.206206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.205998 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:58.207637 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.207615 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" Jul 02 23:28:58.227683 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.227631 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9jfct" podStartSLOduration=2.227616313 podStartE2EDuration="2.227616313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:28:58.225114341 +0000 UTC m=+392.307438265" watchObservedRunningTime="2026-07-02 23:28:58.227616313 +0000 UTC m=+392.309940236" Jul 02 23:28:58.279164 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.279123 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:58.279454 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.279406 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" containerID="cri-o://906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056" gracePeriod=30 Jul 02 23:28:58.280410 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.280390 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:58.574089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.574062 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:58.664922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.664889 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.664939 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.664994 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665112 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665041 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c7sj\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665159 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665196 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665226 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig\") pod \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\" (UID: \"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51\") " Jul 02 23:28:58.665463 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665434 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:28:58.665586 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.665562 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-ca-configmap\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.667909 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.667852 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj" (OuterVolumeSpecName: "kube-api-access-6c7sj") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "kube-api-access-6c7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:28:58.668067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.667991 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts" (OuterVolumeSpecName: "cacerts") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:28:58.668067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.668027 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:28:58.668165 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.668113 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:28:58.668165 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.668114 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs" (OuterVolumeSpecName: "local-certs") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:28:58.668414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.668399 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" (UID: "9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:28:58.766310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766253 2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-kubeconfig\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.766310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766295 2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-local-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.766310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766312 2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-cacerts\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.766542 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766325 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c7sj\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-kube-api-access-6c7sj\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.766542 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766336 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-csr-dns-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:58.766542 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:58.766348 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51-istio-token\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:28:59.211684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.211593 2561 generic.go:356] "Generic (PLEG): container finished" podID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerID="906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056" exitCode=0 Jul 02 23:28:59.211684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.211644 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" Jul 02 23:28:59.212192 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.211678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" event={"ID":"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51","Type":"ContainerDied","Data":"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056"} Jul 02 23:28:59.212192 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.211730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kpszf" event={"ID":"9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51","Type":"ContainerDied","Data":"9b83c21b261897f54c5009906364d0a609f1e4258ce37f79c6fc3db9d387dee6"} Jul 02 23:28:59.212192 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.211749 2561 scope.go:117] "RemoveContainer" containerID="906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056" Jul 02 23:28:59.221654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.221636 2561 scope.go:117] "RemoveContainer" containerID="906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056" Jul 02 23:28:59.221906 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:28:59.221888 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056\": container with ID starting with 906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056 not found: ID does not exist" containerID="906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056" Jul 02 23:28:59.221965 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.221913 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056"} err="failed to get container status \"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056\": rpc error: code = NotFound desc = could not find container \"906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056\": container with ID starting with 906b8ca067ec7e1dd933b5cd60b1f4b01cdebeae922f7fb9d892986e979f3056 not found: ID does not exist" Jul 02 23:28:59.233860 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.233836 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:59.240631 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.240611 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kpszf"] Jul 02 23:28:59.333693 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.333660 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:59.487776 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.487697 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:59.573243 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.573213 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:59.628181 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:28:59.628155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:00.445344 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:00.445311 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" path="/var/lib/kubelet/pods/9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51/volumes" Jul 02 23:29:00.655118 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:00.655086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:00.760255 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:00.760175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:00.858095 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:00.858053 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:06.192967 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:06.192933 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hqbw7" Jul 02 23:29:06.195043 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:06.195024 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-szhhz" Jul 02 23:29:20.234261 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:20.234196 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:20.287473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:20.287442 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:22.184339 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:22.184307 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:31.040239 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:31.040201 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:31.090853 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:31.090821 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:48.871440 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:48.871366 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:48.972276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:48.972238 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:48.999987 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:48.999961 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:49.000474 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.000449 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" Jul 02 23:29:49.000474 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.000474 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" Jul 02 23:29:49.000667 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.000556 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a84fe1f-8e83-4bd8-a34c-64ca6a1fcd51" containerName="discovery" Jul 02 23:29:49.002558 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.002536 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:49.005275 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.005254 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 23:29:49.006272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.006195 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 23:29:49.006272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.006222 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-q546p\"" Jul 02 23:29:49.012696 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.012672 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:49.051743 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.051708 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:29:49.079548 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.079520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psndw\" (UniqueName: \"kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw\") pod \"kuadrant-operator-catalog-ncntn\" (UID: \"65e3b20a-a4cc-4d58-b5ce-d748dca0a953\") " pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:49.180315 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.180223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psndw\" (UniqueName: \"kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw\") pod \"kuadrant-operator-catalog-ncntn\" (UID: \"65e3b20a-a4cc-4d58-b5ce-d748dca0a953\") " pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:49.188448 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.188418 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psndw\" (UniqueName: \"kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw\") pod \"kuadrant-operator-catalog-ncntn\" (UID: \"65e3b20a-a4cc-4d58-b5ce-d748dca0a953\") " pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:49.314414 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.314379 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:49.383972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.383937 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:49.446902 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.446871 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:49.449900 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:29:49.449875 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e3b20a_a4cc_4d58_b5ce_d748dca0a953.slice/crio-9c2ad023632aaafdce7929b6434ca436c2690f71ba76d78ff587254948fc9d01 WatchSource:0}: Error finding container 9c2ad023632aaafdce7929b6434ca436c2690f71ba76d78ff587254948fc9d01: Status 404 returned error can't find the container with id 9c2ad023632aaafdce7929b6434ca436c2690f71ba76d78ff587254948fc9d01 Jul 02 23:29:49.581809 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.581777 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bwmhb"] Jul 02 23:29:49.584634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.584617 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:49.592029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.592005 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bwmhb"] Jul 02 23:29:49.685500 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.685472 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqckp\" (UniqueName: \"kubernetes.io/projected/c77bc0e8-3c14-4a05-b308-6437bd758eea-kube-api-access-sqckp\") pod \"kuadrant-operator-catalog-bwmhb\" (UID: \"c77bc0e8-3c14-4a05-b308-6437bd758eea\") " pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:49.786615 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.786579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqckp\" (UniqueName: \"kubernetes.io/projected/c77bc0e8-3c14-4a05-b308-6437bd758eea-kube-api-access-sqckp\") pod \"kuadrant-operator-catalog-bwmhb\" (UID: \"c77bc0e8-3c14-4a05-b308-6437bd758eea\") " pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:49.794260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.794233 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqckp\" (UniqueName: \"kubernetes.io/projected/c77bc0e8-3c14-4a05-b308-6437bd758eea-kube-api-access-sqckp\") pod \"kuadrant-operator-catalog-bwmhb\" (UID: \"c77bc0e8-3c14-4a05-b308-6437bd758eea\") " pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:49.895635 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:49.895599 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:50.024473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:50.024447 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bwmhb"] Jul 02 23:29:50.025898 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:29:50.025867 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77bc0e8_3c14_4a05_b308_6437bd758eea.slice/crio-51dc3087863786426d17062549f0173217d18737b5aa30d4d61e194c14ebbe3e WatchSource:0}: Error finding container 51dc3087863786426d17062549f0173217d18737b5aa30d4d61e194c14ebbe3e: Status 404 returned error can't find the container with id 51dc3087863786426d17062549f0173217d18737b5aa30d4d61e194c14ebbe3e Jul 02 23:29:50.457660 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:50.457625 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" event={"ID":"c77bc0e8-3c14-4a05-b308-6437bd758eea","Type":"ContainerStarted","Data":"51dc3087863786426d17062549f0173217d18737b5aa30d4d61e194c14ebbe3e"} Jul 02 23:29:50.458847 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:50.458824 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" event={"ID":"65e3b20a-a4cc-4d58-b5ce-d748dca0a953","Type":"ContainerStarted","Data":"9c2ad023632aaafdce7929b6434ca436c2690f71ba76d78ff587254948fc9d01"} Jul 02 23:29:52.471430 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.471392 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" podUID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" containerName="registry-server" containerID="cri-o://749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42" gracePeriod=2 Jul 02 23:29:52.471925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.471390 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" event={"ID":"65e3b20a-a4cc-4d58-b5ce-d748dca0a953","Type":"ContainerStarted","Data":"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42"} Jul 02 23:29:52.473067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.473043 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" event={"ID":"c77bc0e8-3c14-4a05-b308-6437bd758eea","Type":"ContainerStarted","Data":"5c3bb5f13373619955bacf9a4005f1e84da65e50b15f1b14ee38b1ebf2c6731f"} Jul 02 23:29:52.489531 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.489483 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" podStartSLOduration=2.003511735 podStartE2EDuration="4.489464555s" totalImagesPullingTime="2.48595282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:29:52.486375001 +0000 UTC m=+446.568698945" watchObservedRunningTime="2026-07-02 23:29:52.489464555 +0000 UTC m=+446.571788481" Jul 02 23:29:52.500873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.500832 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" podStartSLOduration=1.586648107 podStartE2EDuration="3.500817823s" totalImagesPullingTime="1.914169716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:29:52.499150757 +0000 UTC m=+446.581474684" watchObservedRunningTime="2026-07-02 23:29:52.500817823 +0000 UTC m=+446.583141747" Jul 02 23:29:52.708793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.708763 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:52.811920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.811887 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psndw\" (UniqueName: \"kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw\") pod \"65e3b20a-a4cc-4d58-b5ce-d748dca0a953\" (UID: \"65e3b20a-a4cc-4d58-b5ce-d748dca0a953\") " Jul 02 23:29:52.814123 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.814096 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw" (OuterVolumeSpecName: "kube-api-access-psndw") pod "65e3b20a-a4cc-4d58-b5ce-d748dca0a953" (UID: "65e3b20a-a4cc-4d58-b5ce-d748dca0a953"). InnerVolumeSpecName "kube-api-access-psndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:29:52.912679 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:52.912643 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psndw\" (UniqueName: \"kubernetes.io/projected/65e3b20a-a4cc-4d58-b5ce-d748dca0a953-kube-api-access-psndw\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:29:53.479206 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.479166 2561 generic.go:356] "Generic (PLEG): container finished" podID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" containerID="749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42" exitCode=0 Jul 02 23:29:53.479621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.479211 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" Jul 02 23:29:53.479621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.479265 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" event={"ID":"65e3b20a-a4cc-4d58-b5ce-d748dca0a953","Type":"ContainerDied","Data":"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42"} Jul 02 23:29:53.479621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.479331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ncntn" event={"ID":"65e3b20a-a4cc-4d58-b5ce-d748dca0a953","Type":"ContainerDied","Data":"9c2ad023632aaafdce7929b6434ca436c2690f71ba76d78ff587254948fc9d01"} Jul 02 23:29:53.479621 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.479356 2561 scope.go:117] "RemoveContainer" containerID="749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42" Jul 02 23:29:53.488818 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.488800 2561 scope.go:117] "RemoveContainer" containerID="749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42" Jul 02 23:29:53.489066 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:29:53.489048 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42\": container with ID starting with 749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42 not found: ID does not exist" containerID="749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42" Jul 02 23:29:53.489116 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.489076 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42"} err="failed to get container status \"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42\": rpc error: code = NotFound desc = could not find container \"749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42\": container with ID starting with 749a4fe6f7648d7fc1873790975c3b311792aab54311a375b45db2c1c1245a42 not found: ID does not exist" Jul 02 23:29:53.499615 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.499585 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:53.503230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:53.503211 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ncntn"] Jul 02 23:29:54.445447 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:54.445415 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" path="/var/lib/kubelet/pods/65e3b20a-a4cc-4d58-b5ce-d748dca0a953/volumes" Jul 02 23:29:59.896129 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:59.896092 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:59.896684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:59.896150 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:29:59.919189 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:29:59.919161 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:30:00.532687 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:00.532651 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bwmhb" Jul 02 23:30:04.773639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.773609 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d"] Jul 02 23:30:04.774006 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.773921 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" containerName="registry-server" Jul 02 23:30:04.774006 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.773933 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" containerName="registry-server" Jul 02 23:30:04.774075 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.774011 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e3b20a-a4cc-4d58-b5ce-d748dca0a953" containerName="registry-server" Jul 02 23:30:04.780662 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.777774 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:04.781169 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.781141 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-svhmc\"" Jul 02 23:30:04.788141 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.788118 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d"] Jul 02 23:30:04.912697 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.912667 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpfl\" (UniqueName: \"kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:04.912896 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.912708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:04.912896 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:04.912855 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.014156 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.014118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpfl\" (UniqueName: \"kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.014384 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.014170 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.014384 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.014257 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.014665 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.014641 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.014706 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.014655 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.021688 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.021660 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpfl\" (UniqueName: \"kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.090326 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.090229 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:05.221727 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.221703 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d"] Jul 02 23:30:05.224191 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:05.224170 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f63806f_140b_41c9_bf27_c5d72c038aa3.slice/crio-7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13 WatchSource:0}: Error finding container 7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13: Status 404 returned error can't find the container with id 7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13 Jul 02 23:30:05.535789 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.535743 2561 generic.go:356] "Generic (PLEG): container finished" podID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerID="51d69df7cb47a6f5497c15fd48c6a1d14b2a3a256c7749148eec962d164d6d69" exitCode=0 Jul 02 23:30:05.535939 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.535831 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" event={"ID":"1f63806f-140b-41c9-bf27-c5d72c038aa3","Type":"ContainerDied","Data":"51d69df7cb47a6f5497c15fd48c6a1d14b2a3a256c7749148eec962d164d6d69"} Jul 02 23:30:05.535999 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.535940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" event={"ID":"1f63806f-140b-41c9-bf27-c5d72c038aa3","Type":"ContainerStarted","Data":"7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13"} Jul 02 23:30:05.578620 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.578586 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn"] Jul 02 23:30:05.581197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.581181 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.590439 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.590418 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn"] Jul 02 23:30:05.720066 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.720031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s76\" (UniqueName: \"kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.720250 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.720152 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.720250 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.720190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.820932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.820828 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.820932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.820878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.820932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.820927 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s76\" (UniqueName: \"kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.821495 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.821330 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.821495 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.821357 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.828668 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.828645 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s76\" (UniqueName: \"kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:05.891272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:05.891235 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:06.019238 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.019212 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn"] Jul 02 23:30:06.021344 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:06.021312 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f11ca3c_5950_404f_a30c_45c8177a743d.slice/crio-7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd WatchSource:0}: Error finding container 7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd: Status 404 returned error can't find the container with id 7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd Jul 02 23:30:06.171691 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.171660 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh"] Jul 02 23:30:06.174170 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.174154 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.183069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.183042 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh"] Jul 02 23:30:06.323825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.323800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hkw\" (UniqueName: \"kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.323947 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.323847 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.323947 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.323921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.424331 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.424299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.424521 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.424364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hkw\" (UniqueName: \"kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.424521 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.424453 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.424660 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.424639 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.424845 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.424826 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.432104 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.432072 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hkw\" (UniqueName: \"kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.485238 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.485209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:06.545665 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.545628 2561 generic.go:356] "Generic (PLEG): container finished" podID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerID="292b0a689a78f62ec74e30ca54a253857be4951b26f6205ce204568bf81e6cfd" exitCode=0 Jul 02 23:30:06.545807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.545716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" event={"ID":"4f11ca3c-5950-404f-a30c-45c8177a743d","Type":"ContainerDied","Data":"292b0a689a78f62ec74e30ca54a253857be4951b26f6205ce204568bf81e6cfd"} Jul 02 23:30:06.545807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.545779 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" event={"ID":"4f11ca3c-5950-404f-a30c-45c8177a743d","Type":"ContainerStarted","Data":"7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd"} Jul 02 23:30:06.547623 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.547605 2561 generic.go:356] "Generic (PLEG): container finished" podID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerID="c0e0617cb33d4a851c8d6fcc6a0cae110c14b31d697a4b003b2a850289163a64" exitCode=0 Jul 02 23:30:06.547782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.547643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" event={"ID":"1f63806f-140b-41c9-bf27-c5d72c038aa3","Type":"ContainerDied","Data":"c0e0617cb33d4a851c8d6fcc6a0cae110c14b31d697a4b003b2a850289163a64"} Jul 02 23:30:06.586173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.586132 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2"] Jul 02 23:30:06.589310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.589275 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.598654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.598629 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2"] Jul 02 23:30:06.629450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.629423 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh"] Jul 02 23:30:06.631506 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:06.631479 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e471fe_1eb4_4c7f_9641_1a8b36f7b8ee.slice/crio-b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65 WatchSource:0}: Error finding container b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65: Status 404 returned error can't find the container with id b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65 Jul 02 23:30:06.726238 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.726202 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.726373 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.726323 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.726373 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.726362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ch7v\" (UniqueName: \"kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.827479 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.827407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.827479 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.827452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ch7v\" (UniqueName: \"kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.827873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.827524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.827873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.827752 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.827873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.827817 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.835223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.835200 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ch7v\" (UniqueName: \"kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:06.905241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:06.905214 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:07.034973 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.034947 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2"] Jul 02 23:30:07.039132 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:07.038624 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc67af1_61af_4328_b581_e93c82ac1ace.slice/crio-52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91 WatchSource:0}: Error finding container 52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91: Status 404 returned error can't find the container with id 52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91 Jul 02 23:30:07.555171 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.555131 2561 generic.go:356] "Generic (PLEG): container finished" podID="abc67af1-61af-4328-b581-e93c82ac1ace" containerID="66fe6daaed2e4bcbf57dfbf4c14ce85218263e080ff1727da3349b55da3d341b" exitCode=0 Jul 02 23:30:07.555380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.555216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" event={"ID":"abc67af1-61af-4328-b581-e93c82ac1ace","Type":"ContainerDied","Data":"66fe6daaed2e4bcbf57dfbf4c14ce85218263e080ff1727da3349b55da3d341b"} Jul 02 23:30:07.555380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.555324 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" event={"ID":"abc67af1-61af-4328-b581-e93c82ac1ace","Type":"ContainerStarted","Data":"52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91"} Jul 02 23:30:07.557481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.557453 2561 generic.go:356] "Generic (PLEG): container finished" podID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerID="7a9c1514c44877bd6cb79fff3fa3755168b106a56a6356e91cd18511b6b6720e" exitCode=0 Jul 02 23:30:07.557594 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.557520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" event={"ID":"1f63806f-140b-41c9-bf27-c5d72c038aa3","Type":"ContainerDied","Data":"7a9c1514c44877bd6cb79fff3fa3755168b106a56a6356e91cd18511b6b6720e"} Jul 02 23:30:07.559075 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.559052 2561 generic.go:356] "Generic (PLEG): container finished" podID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerID="d737024069abb18a839835203a868d4f8dc4ec9758f57920855f003ad5549939" exitCode=0 Jul 02 23:30:07.559187 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.559134 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" event={"ID":"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee","Type":"ContainerDied","Data":"d737024069abb18a839835203a868d4f8dc4ec9758f57920855f003ad5549939"} Jul 02 23:30:07.559187 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.559178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" event={"ID":"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee","Type":"ContainerStarted","Data":"b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65"} Jul 02 23:30:07.561158 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.561138 2561 generic.go:356] "Generic (PLEG): container finished" podID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerID="66386130f3e3ce794d34bb0379ccf5091be339f8766b7dc34167e97c1b4892f8" exitCode=0 Jul 02 23:30:07.561276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:07.561173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" event={"ID":"4f11ca3c-5950-404f-a30c-45c8177a743d","Type":"ContainerDied","Data":"66386130f3e3ce794d34bb0379ccf5091be339f8766b7dc34167e97c1b4892f8"} Jul 02 23:30:08.570887 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.570840 2561 generic.go:356] "Generic (PLEG): container finished" podID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerID="6953fd4eee1594caded2a58b2b25050f8380a2dfb4a830f2194bdaa727a1c34d" exitCode=0 Jul 02 23:30:08.571300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.570923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" event={"ID":"4f11ca3c-5950-404f-a30c-45c8177a743d","Type":"ContainerDied","Data":"6953fd4eee1594caded2a58b2b25050f8380a2dfb4a830f2194bdaa727a1c34d"} Jul 02 23:30:08.572642 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.572620 2561 generic.go:356] "Generic (PLEG): container finished" podID="abc67af1-61af-4328-b581-e93c82ac1ace" containerID="b429ca76f2040b0ee93b0fc5a75b60f50fed6abc2d4f95d0fa1aa707624d1677" exitCode=0 Jul 02 23:30:08.572768 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.572704 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" event={"ID":"abc67af1-61af-4328-b581-e93c82ac1ace","Type":"ContainerDied","Data":"b429ca76f2040b0ee93b0fc5a75b60f50fed6abc2d4f95d0fa1aa707624d1677"} Jul 02 23:30:08.574562 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.574539 2561 generic.go:356] "Generic (PLEG): container finished" podID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerID="26a2bf65d448c61779b11a8e0c73eeb1edaab0bd29864caeb427c3bceaaa8ab7" exitCode=0 Jul 02 23:30:08.574659 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.574569 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" event={"ID":"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee","Type":"ContainerDied","Data":"26a2bf65d448c61779b11a8e0c73eeb1edaab0bd29864caeb427c3bceaaa8ab7"} Jul 02 23:30:08.704220 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.704152 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:08.848023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.847997 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqpfl\" (UniqueName: \"kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl\") pod \"1f63806f-140b-41c9-bf27-c5d72c038aa3\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " Jul 02 23:30:08.848023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.848028 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util\") pod \"1f63806f-140b-41c9-bf27-c5d72c038aa3\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " Jul 02 23:30:08.848195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.848116 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle\") pod \"1f63806f-140b-41c9-bf27-c5d72c038aa3\" (UID: \"1f63806f-140b-41c9-bf27-c5d72c038aa3\") " Jul 02 23:30:08.848519 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.848497 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle" (OuterVolumeSpecName: "bundle") pod "1f63806f-140b-41c9-bf27-c5d72c038aa3" (UID: "1f63806f-140b-41c9-bf27-c5d72c038aa3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:08.850183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.850160 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl" (OuterVolumeSpecName: "kube-api-access-kqpfl") pod "1f63806f-140b-41c9-bf27-c5d72c038aa3" (UID: "1f63806f-140b-41c9-bf27-c5d72c038aa3"). InnerVolumeSpecName "kube-api-access-kqpfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:08.853509 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.853489 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util" (OuterVolumeSpecName: "util") pod "1f63806f-140b-41c9-bf27-c5d72c038aa3" (UID: "1f63806f-140b-41c9-bf27-c5d72c038aa3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:08.949260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.949175 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpfl\" (UniqueName: \"kubernetes.io/projected/1f63806f-140b-41c9-bf27-c5d72c038aa3-kube-api-access-kqpfl\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:08.949260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.949212 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:08.949260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:08.949222 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f63806f-140b-41c9-bf27-c5d72c038aa3-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:09.582041 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.582009 2561 generic.go:356] "Generic (PLEG): container finished" podID="abc67af1-61af-4328-b581-e93c82ac1ace" containerID="6fc70ebbfceabab17e0fb99521ee52ebd737dad4fcdb9b0a5399399d03ee00c1" exitCode=0 Jul 02 23:30:09.582445 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.582089 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" event={"ID":"abc67af1-61af-4328-b581-e93c82ac1ace","Type":"ContainerDied","Data":"6fc70ebbfceabab17e0fb99521ee52ebd737dad4fcdb9b0a5399399d03ee00c1"} Jul 02 23:30:09.583925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.583910 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" Jul 02 23:30:09.583925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.583915 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d" event={"ID":"1f63806f-140b-41c9-bf27-c5d72c038aa3","Type":"ContainerDied","Data":"7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13"} Jul 02 23:30:09.584095 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.583953 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4ff74b195962a2cc0e4d779ebb5c523caaffb3baed00cf1691d0387eb7dd13" Jul 02 23:30:09.586070 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.586040 2561 generic.go:356] "Generic (PLEG): container finished" podID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerID="d8639bea47fb764824f408436d8ee62472fb38c4c5ef14e5acc7c6a209afb2cb" exitCode=0 Jul 02 23:30:09.586203 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.586128 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" event={"ID":"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee","Type":"ContainerDied","Data":"d8639bea47fb764824f408436d8ee62472fb38c4c5ef14e5acc7c6a209afb2cb"} Jul 02 23:30:09.714412 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.714386 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:09.855944 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.855847 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle\") pod \"4f11ca3c-5950-404f-a30c-45c8177a743d\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " Jul 02 23:30:09.855944 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.855932 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util\") pod \"4f11ca3c-5950-404f-a30c-45c8177a743d\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " Jul 02 23:30:09.856189 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.855982 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6s76\" (UniqueName: \"kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76\") pod \"4f11ca3c-5950-404f-a30c-45c8177a743d\" (UID: \"4f11ca3c-5950-404f-a30c-45c8177a743d\") " Jul 02 23:30:09.856524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.856494 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle" (OuterVolumeSpecName: "bundle") pod "4f11ca3c-5950-404f-a30c-45c8177a743d" (UID: "4f11ca3c-5950-404f-a30c-45c8177a743d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:09.858217 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.858194 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76" (OuterVolumeSpecName: "kube-api-access-z6s76") pod "4f11ca3c-5950-404f-a30c-45c8177a743d" (UID: "4f11ca3c-5950-404f-a30c-45c8177a743d"). InnerVolumeSpecName "kube-api-access-z6s76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:09.861393 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.861355 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util" (OuterVolumeSpecName: "util") pod "4f11ca3c-5950-404f-a30c-45c8177a743d" (UID: "4f11ca3c-5950-404f-a30c-45c8177a743d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:09.956950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.956905 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:09.957099 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.956966 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6s76\" (UniqueName: \"kubernetes.io/projected/4f11ca3c-5950-404f-a30c-45c8177a743d-kube-api-access-z6s76\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:09.957099 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:09.956981 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f11ca3c-5950-404f-a30c-45c8177a743d-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.593408 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.593382 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" Jul 02 23:30:10.593832 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.593408 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn" event={"ID":"4f11ca3c-5950-404f-a30c-45c8177a743d","Type":"ContainerDied","Data":"7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd"} Jul 02 23:30:10.593832 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.593461 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcd3c3af2c2ef621cdacc47f1c989dd2ff8e6566201c97519744b5d4458e1dd" Jul 02 23:30:10.756668 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.756646 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:10.765826 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.765806 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:10.865837 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.865805 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2hkw\" (UniqueName: \"kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw\") pod \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " Jul 02 23:30:10.866029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.865859 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle\") pod \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " Jul 02 23:30:10.866029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.865907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util\") pod \"abc67af1-61af-4328-b581-e93c82ac1ace\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " Jul 02 23:30:10.866029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.865953 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util\") pod \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\" (UID: \"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee\") " Jul 02 23:30:10.866029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.865978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle\") pod \"abc67af1-61af-4328-b581-e93c82ac1ace\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " Jul 02 23:30:10.866029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.866022 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ch7v\" (UniqueName: \"kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v\") pod \"abc67af1-61af-4328-b581-e93c82ac1ace\" (UID: \"abc67af1-61af-4328-b581-e93c82ac1ace\") " Jul 02 23:30:10.866691 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.866636 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle" (OuterVolumeSpecName: "bundle") pod "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" (UID: "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:10.866783 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.866767 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle" (OuterVolumeSpecName: "bundle") pod "abc67af1-61af-4328-b581-e93c82ac1ace" (UID: "abc67af1-61af-4328-b581-e93c82ac1ace"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:10.868209 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.868184 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw" (OuterVolumeSpecName: "kube-api-access-m2hkw") pod "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" (UID: "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee"). InnerVolumeSpecName "kube-api-access-m2hkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:10.868629 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.868606 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v" (OuterVolumeSpecName: "kube-api-access-6ch7v") pod "abc67af1-61af-4328-b581-e93c82ac1ace" (UID: "abc67af1-61af-4328-b581-e93c82ac1ace"). InnerVolumeSpecName "kube-api-access-6ch7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:10.871599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.871576 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util" (OuterVolumeSpecName: "util") pod "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" (UID: "92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:10.872630 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.872610 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util" (OuterVolumeSpecName: "util") pod "abc67af1-61af-4328-b581-e93c82ac1ace" (UID: "abc67af1-61af-4328-b581-e93c82ac1ace"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:10.966795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966763 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ch7v\" (UniqueName: \"kubernetes.io/projected/abc67af1-61af-4328-b581-e93c82ac1ace-kube-api-access-6ch7v\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.966795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966790 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hkw\" (UniqueName: \"kubernetes.io/projected/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-kube-api-access-m2hkw\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.966795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966799 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.967026 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966809 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.967026 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966817 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee-util\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:10.967026 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:10.966824 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abc67af1-61af-4328-b581-e93c82ac1ace-bundle\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:11.600578 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.600550 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" Jul 02 23:30:11.600978 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.600550 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2" event={"ID":"abc67af1-61af-4328-b581-e93c82ac1ace","Type":"ContainerDied","Data":"52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91"} Jul 02 23:30:11.600978 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.600678 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52329dc6f719dc3d21e5c470f474a8201b9062921427e25aa873a9b9acfb8b91" Jul 02 23:30:11.602497 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.602469 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" Jul 02 23:30:11.602497 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.602468 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh" event={"ID":"92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee","Type":"ContainerDied","Data":"b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65"} Jul 02 23:30:11.602655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:11.602508 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ab2d6b07853037183a21af6fd4ae917c76f17640d4af6721347af87d31bb65" Jul 02 23:30:16.914494 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:16.914456 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:16.967481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:16.967442 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:18.494113 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:18.494076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:20.721170 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721130 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt"] Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721469 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721482 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721498 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721503 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721511 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721516 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721522 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721527 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721534 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721539 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721546 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721551 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="util" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721559 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721564 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721572 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="extract" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721577 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="extract" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721585 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="extract" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721589 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="extract" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721594 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721599 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="pull" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721607 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="extract" Jul 02 23:30:20.721601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721612 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721618 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721623 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721672 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f63806f-140b-41c9-bf27-c5d72c038aa3" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721680 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f11ca3c-5950-404f-a30c-45c8177a743d" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721686 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc67af1-61af-4328-b581-e93c82ac1ace" containerName="extract" Jul 02 23:30:20.722335 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.721694 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee" containerName="extract" Jul 02 23:30:20.724718 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.724702 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:20.727798 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.727776 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qf89b\"" Jul 02 23:30:20.737035 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.737008 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt"] Jul 02 23:30:20.853503 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.853465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkkh7\" (UniqueName: \"kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" (UID: \"40071949-1e6a-45f1-9838-0d47b3befe70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:20.954847 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.954806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkkh7\" (UniqueName: \"kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" (UID: \"40071949-1e6a-45f1-9838-0d47b3befe70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:20.971951 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:20.971882 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkkh7\" (UniqueName: \"kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" (UID: \"40071949-1e6a-45f1-9838-0d47b3befe70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:21.035502 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:21.035468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:21.174321 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:21.174272 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt"] Jul 02 23:30:21.176373 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:21.176341 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40071949_1e6a_45f1_9838_0d47b3befe70.slice/crio-e2c5d1a7a286540111c40755b18c75508cafe403e872c6b5e0335690d2530e22 WatchSource:0}: Error finding container e2c5d1a7a286540111c40755b18c75508cafe403e872c6b5e0335690d2530e22: Status 404 returned error can't find the container with id e2c5d1a7a286540111c40755b18c75508cafe403e872c6b5e0335690d2530e22 Jul 02 23:30:21.529591 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:21.529554 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:21.654164 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:21.654132 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" event={"ID":"40071949-1e6a-45f1-9838-0d47b3befe70","Type":"ContainerStarted","Data":"e2c5d1a7a286540111c40755b18c75508cafe403e872c6b5e0335690d2530e22"} Jul 02 23:30:22.289142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:22.289082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:23.666305 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:23.666244 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" event={"ID":"40071949-1e6a-45f1-9838-0d47b3befe70","Type":"ContainerStarted","Data":"8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed"} Jul 02 23:30:23.666712 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:23.666357 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:23.696951 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:23.696897 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" podStartSLOduration=1.869536705 podStartE2EDuration="3.696879766s" totalImagesPullingTime="1.827343061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:23.694671474 +0000 UTC m=+477.776995398" watchObservedRunningTime="2026-07-02 23:30:23.696879766 +0000 UTC m=+477.779203690" Jul 02 23:30:25.939182 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:25.939144 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:26.126599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:26.126563 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:27.710771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.710741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:27.829556 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.829514 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf7d699f9-gq8xp"] Jul 02 23:30:27.832606 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.832587 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.834963 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.834939 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 23:30:27.835091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.835030 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 23:30:27.835187 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.835164 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 23:30:27.836232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.836213 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 23:30:27.836374 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.836335 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 23:30:27.836434 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.836420 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 23:30:27.836562 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.836541 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 23:30:27.837947 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.837923 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fqzxt\"" Jul 02 23:30:27.842544 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.842522 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 23:30:27.843326 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.843302 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf7d699f9-gq8xp"] Jul 02 23:30:27.886012 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.885968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:27.904340 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-oauth-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-oauth-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txp82\" (UniqueName: \"kubernetes.io/projected/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-kube-api-access-txp82\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904427 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-service-ca\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904455 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-trusted-ca-bundle\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:27.904518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:27.904475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005109 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-oauth-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005173 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-oauth-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txp82\" (UniqueName: \"kubernetes.io/projected/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-kube-api-access-txp82\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005554 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005222 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-service-ca\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005554 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005249 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-trusted-ca-bundle\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005554 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005278 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.005989 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005966 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-service-ca\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.006099 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.005983 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-oauth-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.006226 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.006208 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-trusted-ca-bundle\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.006261 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.006212 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.007922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.007901 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-serving-cert\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.008005 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.007929 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-console-oauth-config\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.014743 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.014714 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txp82\" (UniqueName: \"kubernetes.io/projected/751e6bd0-5c22-44eb-bf6f-77708fd8fbab-kube-api-access-txp82\") pod \"console-7cf7d699f9-gq8xp\" (UID: \"751e6bd0-5c22-44eb-bf6f-77708fd8fbab\") " pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.130471 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.130433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:28.140179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.140149 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d"] Jul 02 23:30:28.143123 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.143107 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:28.144167 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.144150 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:28.146547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.146531 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9p5t7\"" Jul 02 23:30:28.146795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.146779 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 02 23:30:28.164497 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.164466 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d"] Jul 02 23:30:28.206393 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.206356 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zz7\" (UniqueName: \"kubernetes.io/projected/109e0945-51a7-46fa-abbe-af71229f0aea-kube-api-access-v5zz7\") pod \"dns-operator-controller-manager-864dd78c4c-2nw4d\" (UID: \"109e0945-51a7-46fa-abbe-af71229f0aea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:28.235255 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.235216 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:28.289771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.289744 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf7d699f9-gq8xp"] Jul 02 23:30:28.291385 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:28.291351 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751e6bd0_5c22_44eb_bf6f_77708fd8fbab.slice/crio-20aa295aa6a945b13216fd19788ad43078d7b49d581c191b4825f5536aa6f7dd WatchSource:0}: Error finding container 20aa295aa6a945b13216fd19788ad43078d7b49d581c191b4825f5536aa6f7dd: Status 404 returned error can't find the container with id 20aa295aa6a945b13216fd19788ad43078d7b49d581c191b4825f5536aa6f7dd Jul 02 23:30:28.307536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.307514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zz7\" (UniqueName: \"kubernetes.io/projected/109e0945-51a7-46fa-abbe-af71229f0aea-kube-api-access-v5zz7\") pod \"dns-operator-controller-manager-864dd78c4c-2nw4d\" (UID: \"109e0945-51a7-46fa-abbe-af71229f0aea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:28.321047 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.321022 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zz7\" (UniqueName: \"kubernetes.io/projected/109e0945-51a7-46fa-abbe-af71229f0aea-kube-api-access-v5zz7\") pod \"dns-operator-controller-manager-864dd78c4c-2nw4d\" (UID: \"109e0945-51a7-46fa-abbe-af71229f0aea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:28.454481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.454454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:28.595656 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.595628 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d"] Jul 02 23:30:28.597349 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:28.597323 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109e0945_51a7_46fa_abbe_af71229f0aea.slice/crio-2c835f120b3a1c4296a63180076a86f6efa8b740b992a5509d9cf73616a8a032 WatchSource:0}: Error finding container 2c835f120b3a1c4296a63180076a86f6efa8b740b992a5509d9cf73616a8a032: Status 404 returned error can't find the container with id 2c835f120b3a1c4296a63180076a86f6efa8b740b992a5509d9cf73616a8a032 Jul 02 23:30:28.626634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.626607 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:28.693046 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.693012 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf7d699f9-gq8xp" event={"ID":"751e6bd0-5c22-44eb-bf6f-77708fd8fbab","Type":"ContainerStarted","Data":"605ffda11497fed45349feed7838308b9f9c36aab6759035308fde350ecab5f0"} Jul 02 23:30:28.693253 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.693070 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf7d699f9-gq8xp" event={"ID":"751e6bd0-5c22-44eb-bf6f-77708fd8fbab","Type":"ContainerStarted","Data":"20aa295aa6a945b13216fd19788ad43078d7b49d581c191b4825f5536aa6f7dd"} Jul 02 23:30:28.694345 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.694323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" event={"ID":"109e0945-51a7-46fa-abbe-af71229f0aea","Type":"ContainerStarted","Data":"2c835f120b3a1c4296a63180076a86f6efa8b740b992a5509d9cf73616a8a032"} Jul 02 23:30:28.711063 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.710669 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf7d699f9-gq8xp" podStartSLOduration=1.7106503480000002 podStartE2EDuration="1.710650348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:28.709596179 +0000 UTC m=+482.791920107" watchObservedRunningTime="2026-07-02 23:30:28.710650348 +0000 UTC m=+482.792974274" Jul 02 23:30:28.721638 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.721606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:28.905975 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:28.905887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:29.189378 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:29.189276 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:31.167925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:31.167894 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:31.713769 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:31.713731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" event={"ID":"109e0945-51a7-46fa-abbe-af71229f0aea","Type":"ContainerStarted","Data":"a5080dab44ebdc3f76d06bbfd6dab1186c4ee8d701013d9ecb7b44bc91f003c5"} Jul 02 23:30:31.713972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:31.713857 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:31.733131 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:31.733078 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" podStartSLOduration=1.5756667800000002 podStartE2EDuration="3.733063994s" totalImagesPullingTime="2.157397214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:31.730459891 +0000 UTC m=+485.812783817" watchObservedRunningTime="2026-07-02 23:30:31.733063994 +0000 UTC m=+485.815387918" Jul 02 23:30:32.707179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.707144 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl"] Jul 02 23:30:32.710535 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.710520 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.713464 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.713447 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ml5c8\"" Jul 02 23:30:32.730700 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.730672 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl"] Jul 02 23:30:32.743627 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.743581 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.743804 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.743636 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhzj\" (UniqueName: \"kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.844964 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.844918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.845181 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.845007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhzj\" (UniqueName: \"kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.845426 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.845405 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:32.854067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:32.854033 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhzj\" (UniqueName: \"kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:33.026260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:33.026227 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:33.173520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:33.173496 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl"] Jul 02 23:30:33.175992 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:33.175958 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1fabea_bc9c_4374_8bc2_2822511a2116.slice/crio-5e5273a68cecb45006aec457d65a35630d2f17319e1f64e6c53e2c8afd3774cb WatchSource:0}: Error finding container 5e5273a68cecb45006aec457d65a35630d2f17319e1f64e6c53e2c8afd3774cb: Status 404 returned error can't find the container with id 5e5273a68cecb45006aec457d65a35630d2f17319e1f64e6c53e2c8afd3774cb Jul 02 23:30:33.728223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:33.728188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" event={"ID":"7e1fabea-bc9c-4374-8bc2-2822511a2116","Type":"ContainerStarted","Data":"5e5273a68cecb45006aec457d65a35630d2f17319e1f64e6c53e2c8afd3774cb"} Jul 02 23:30:34.095692 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:34.095661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:34.196242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:34.196209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:34.674048 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:34.674005 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:38.144705 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.144668 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:38.145155 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.144775 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:38.150413 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.150389 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:38.760033 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.759993 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" event={"ID":"7e1fabea-bc9c-4374-8bc2-2822511a2116","Type":"ContainerStarted","Data":"11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79"} Jul 02 23:30:38.764066 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.764044 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf7d699f9-gq8xp" Jul 02 23:30:38.778827 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.778780 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" podStartSLOduration=1.345133067 podStartE2EDuration="6.7787666s" totalImagesPullingTime="5.433633533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:38.776334218 +0000 UTC m=+492.858658146" watchObservedRunningTime="2026-07-02 23:30:38.7787666 +0000 UTC m=+492.861090523" Jul 02 23:30:38.914135 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:38.914065 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:39.014842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:39.014815 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:39.490529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:39.490485 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:39.539852 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:39.539823 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:39.765887 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:39.765863 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:39.980743 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:39.980709 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:40.095934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:40.095844 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:41.225238 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:41.225209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:41.740446 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:41.740407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:42.724829 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:42.724795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nw4d" Jul 02 23:30:42.904146 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:42.904115 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:43.007477 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:43.007392 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:45.188747 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:45.188712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:45.236600 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:45.236569 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:47.016950 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:47.016914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:47.115983 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:47.115951 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:50.774253 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:50.774220 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.452477 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.452450 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl"] Jul 02 23:30:52.452906 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.452622 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" containerName="manager" containerID="cri-o://11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79" gracePeriod=2 Jul 02 23:30:52.465418 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.465389 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl"] Jul 02 23:30:52.467879 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.467839 2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7e1fabea-bc9c-4374-8bc2-2822511a2116\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\": pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.475673 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.475645 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:30:52.476062 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.476047 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" containerName="manager" Jul 02 23:30:52.476062 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.476063 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" containerName="manager" Jul 02 23:30:52.476191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.476138 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" containerName="manager" Jul 02 23:30:52.479216 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.479195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.481322 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.481276 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.482169 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.482143 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt"] Jul 02 23:30:52.482377 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.482354 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" containerName="manager" containerID="cri-o://8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed" gracePeriod=2 Jul 02 23:30:52.488040 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.487992 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt"] Jul 02 23:30:52.491798 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.491774 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:30:52.501084 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.501060 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x"] Jul 02 23:30:52.501434 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.501420 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" containerName="manager" Jul 02 23:30:52.501509 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.501436 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" containerName="manager" Jul 02 23:30:52.501570 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.501522 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" containerName="manager" Jul 02 23:30:52.504456 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.504438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:52.513649 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.513617 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.516525 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.516494 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.517050 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.517026 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x"] Jul 02 23:30:52.619920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.619638 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9x6\" (UniqueName: \"kubernetes.io/projected/6737be2e-c7aa-42ff-b145-bc524e6ae53b-kube-api-access-wq9x6\") pod \"limitador-operator-controller-manager-859d7b78bd-gc44x\" (UID: \"6737be2e-c7aa-42ff-b145-bc524e6ae53b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:52.619920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.619754 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxzd6\" (UniqueName: \"kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.619920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.619832 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.720741 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.720710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.720855 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.720754 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9x6\" (UniqueName: \"kubernetes.io/projected/6737be2e-c7aa-42ff-b145-bc524e6ae53b-kube-api-access-wq9x6\") pod \"limitador-operator-controller-manager-859d7b78bd-gc44x\" (UID: \"6737be2e-c7aa-42ff-b145-bc524e6ae53b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:52.720855 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.720798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxzd6\" (UniqueName: \"kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.721132 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.721110 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.727626 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.727598 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.729559 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.729529 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.730513 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.730495 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.730891 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.730875 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9x6\" (UniqueName: \"kubernetes.io/projected/6737be2e-c7aa-42ff-b145-bc524e6ae53b-kube-api-access-wq9x6\") pod \"limitador-operator-controller-manager-859d7b78bd-gc44x\" (UID: \"6737be2e-c7aa-42ff-b145-bc524e6ae53b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:52.730994 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.730977 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxzd6\" (UniqueName: \"kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lp9w4\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.731199 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.731180 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.732789 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.732771 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.734351 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.734325 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.821125 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.821091 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkkh7\" (UniqueName: \"kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7\") pod \"40071949-1e6a-45f1-9838-0d47b3befe70\" (UID: \"40071949-1e6a-45f1-9838-0d47b3befe70\") " Jul 02 23:30:52.821125 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.821129 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume\") pod \"7e1fabea-bc9c-4374-8bc2-2822511a2116\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " Jul 02 23:30:52.821395 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.821155 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjhzj\" (UniqueName: \"kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj\") pod \"7e1fabea-bc9c-4374-8bc2-2822511a2116\" (UID: \"7e1fabea-bc9c-4374-8bc2-2822511a2116\") " Jul 02 23:30:52.821735 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.821704 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7e1fabea-bc9c-4374-8bc2-2822511a2116" (UID: "7e1fabea-bc9c-4374-8bc2-2822511a2116"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:30:52.823429 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.823405 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7" (OuterVolumeSpecName: "kube-api-access-lkkh7") pod "40071949-1e6a-45f1-9838-0d47b3befe70" (UID: "40071949-1e6a-45f1-9838-0d47b3befe70"). InnerVolumeSpecName "kube-api-access-lkkh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:52.823518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.823449 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj" (OuterVolumeSpecName: "kube-api-access-cjhzj") pod "7e1fabea-bc9c-4374-8bc2-2822511a2116" (UID: "7e1fabea-bc9c-4374-8bc2-2822511a2116"). InnerVolumeSpecName "kube-api-access-cjhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:52.843571 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.843543 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e1fabea-bc9c-4374-8bc2-2822511a2116" containerID="11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79" exitCode=0 Jul 02 23:30:52.843683 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.843583 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.843683 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.843645 2561 scope.go:117] "RemoveContainer" containerID="11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79" Jul 02 23:30:52.845130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.845108 2561 generic.go:356] "Generic (PLEG): container finished" podID="40071949-1e6a-45f1-9838-0d47b3befe70" containerID="8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed" exitCode=0 Jul 02 23:30:52.845211 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.845148 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.845765 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.845737 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.847553 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.847529 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.849375 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.849347 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.851220 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.851193 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.853841 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.853824 2561 scope.go:117] "RemoveContainer" containerID="11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79" Jul 02 23:30:52.854101 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.854079 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79\": container with ID starting with 11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79 not found: ID does not exist" containerID="11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79" Jul 02 23:30:52.854178 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.854108 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79"} err="failed to get container status \"11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79\": rpc error: code = NotFound desc = could not find container \"11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79\": container with ID starting with 11c172f992c711856f85abe803fcecae2cd884e6aa757b9f2d51d9adf8299d79 not found: ID does not exist" Jul 02 23:30:52.854178 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.854123 2561 scope.go:117] "RemoveContainer" containerID="8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed" Jul 02 23:30:52.857841 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.857812 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.859625 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.859602 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.861430 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.861409 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:52.862172 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.862156 2561 scope.go:117] "RemoveContainer" containerID="8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed" Jul 02 23:30:52.862466 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.862446 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed\": container with ID starting with 8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed not found: ID does not exist" containerID="8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed" Jul 02 23:30:52.862537 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.862478 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed"} err="failed to get container status \"8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed\": rpc error: code = NotFound desc = could not find container \"8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed\": container with ID starting with 8c0ea7ec1c44ed81f095d6174db21cb3c94b6f9c42e14aa4e0532150e315a6ed not found: ID does not exist" Jul 02 23:30:52.863150 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:52.863133 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:52.888067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.888048 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:52.895240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.895223 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:52.922559 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.922530 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkkh7\" (UniqueName: \"kubernetes.io/projected/40071949-1e6a-45f1-9838-0d47b3befe70-kube-api-access-lkkh7\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:52.922686 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.922566 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e1fabea-bc9c-4374-8bc2-2822511a2116-extensions-socket-volume\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:52.922686 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:52.922584 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhzj\" (UniqueName: \"kubernetes.io/projected/7e1fabea-bc9c-4374-8bc2-2822511a2116-kube-api-access-cjhzj\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:30:53.039450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.039420 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:30:53.041985 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:53.041939 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517811fb_470a_4f6f_9a76_6da29e9a32ec.slice/crio-6f1fc79b8f8e603e1322b271a44fd8f7dd21f98c19e45a9352918e8cc6893ce1 WatchSource:0}: Error finding container 6f1fc79b8f8e603e1322b271a44fd8f7dd21f98c19e45a9352918e8cc6893ce1: Status 404 returned error can't find the container with id 6f1fc79b8f8e603e1322b271a44fd8f7dd21f98c19e45a9352918e8cc6893ce1 Jul 02 23:30:53.054481 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.054442 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x"] Jul 02 23:30:53.055392 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:30:53.055364 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6737be2e_c7aa_42ff_b145_bc524e6ae53b.slice/crio-8d6f2d3b659dc3cd0b7493724a1753903845159c65f5f08acae276b0ff40e8da WatchSource:0}: Error finding container 8d6f2d3b659dc3cd0b7493724a1753903845159c65f5f08acae276b0ff40e8da: Status 404 returned error can't find the container with id 8d6f2d3b659dc3cd0b7493724a1753903845159c65f5f08acae276b0ff40e8da Jul 02 23:30:53.465011 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.464972 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:53.565448 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.565421 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:53.672365 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.672334 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:53.772859 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.772829 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:53.854046 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.854009 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" event={"ID":"6737be2e-c7aa-42ff-b145-bc524e6ae53b","Type":"ContainerStarted","Data":"a0fe7109561b617af8b4e5c7fb5a415b4f2a52f60bafd853af5aaaed7498437c"} Jul 02 23:30:53.854242 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.854190 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:30:53.854328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.854234 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" event={"ID":"6737be2e-c7aa-42ff-b145-bc524e6ae53b","Type":"ContainerStarted","Data":"8d6f2d3b659dc3cd0b7493724a1753903845159c65f5f08acae276b0ff40e8da"} Jul 02 23:30:53.856635 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.856610 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" event={"ID":"517811fb-470a-4f6f-9a76-6da29e9a32ec","Type":"ContainerStarted","Data":"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426"} Jul 02 23:30:53.856755 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.856654 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" event={"ID":"517811fb-470a-4f6f-9a76-6da29e9a32ec","Type":"ContainerStarted","Data":"6f1fc79b8f8e603e1322b271a44fd8f7dd21f98c19e45a9352918e8cc6893ce1"} Jul 02 23:30:53.856755 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.856723 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:30:53.879539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.879496 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" podStartSLOduration=1.87948242 podStartE2EDuration="1.87948242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:53.87797282 +0000 UTC m=+507.960296759" watchObservedRunningTime="2026-07-02 23:30:53.87948242 +0000 UTC m=+507.961806343" Jul 02 23:30:53.879776 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:53.879752 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:53.881296 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:53.881266 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:53.900693 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:53.900653 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" podStartSLOduration=1.900640808 podStartE2EDuration="1.900640808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:30:53.899136014 +0000 UTC m=+507.981459939" watchObservedRunningTime="2026-07-02 23:30:53.900640808 +0000 UTC m=+507.982964814" Jul 02 23:30:53.900846 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:53.900816 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pgfgt\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pgfgt" Jul 02 23:30:53.902536 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:30:53.902513 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dnsfl\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dnsfl" Jul 02 23:30:54.445617 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:54.445573 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40071949-1e6a-45f1-9838-0d47b3befe70" path="/var/lib/kubelet/pods/40071949-1e6a-45f1-9838-0d47b3befe70/volumes" Jul 02 23:30:54.446024 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:54.446003 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1fabea-bc9c-4374-8bc2-2822511a2116" path="/var/lib/kubelet/pods/7e1fabea-bc9c-4374-8bc2-2822511a2116/volumes" Jul 02 23:30:57.458237 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:57.458198 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:57.521675 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:57.521647 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:57.655588 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:57.655558 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:57.711030 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:30:57.710933 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:01.248829 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:01.248795 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:01.295792 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:01.295764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:04.864747 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:04.864717 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc44x" Jul 02 23:31:04.865270 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:04.864783 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:31:05.799814 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:05.799767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:05.912326 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:05.912273 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:08.559045 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.559007 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:31:08.559554 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.559209 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" podUID="517811fb-470a-4f6f-9a76-6da29e9a32ec" containerName="manager" containerID="cri-o://c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426" gracePeriod=10 Jul 02 23:31:08.731762 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.731674 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:31:08.734820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.734795 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.748179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.748154 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:31:08.819246 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.819193 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:31:08.854205 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.854174 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.854394 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.854264 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvk5\" (UniqueName: \"kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.938400 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.938370 2561 generic.go:356] "Generic (PLEG): container finished" podID="517811fb-470a-4f6f-9a76-6da29e9a32ec" containerID="c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426" exitCode=0 Jul 02 23:31:08.938560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.938413 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" Jul 02 23:31:08.938560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.938442 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" event={"ID":"517811fb-470a-4f6f-9a76-6da29e9a32ec","Type":"ContainerDied","Data":"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426"} Jul 02 23:31:08.938560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.938482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4" event={"ID":"517811fb-470a-4f6f-9a76-6da29e9a32ec","Type":"ContainerDied","Data":"6f1fc79b8f8e603e1322b271a44fd8f7dd21f98c19e45a9352918e8cc6893ce1"} Jul 02 23:31:08.938560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.938505 2561 scope.go:117] "RemoveContainer" containerID="c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426" Jul 02 23:31:08.952612 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.952590 2561 scope.go:117] "RemoveContainer" containerID="c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426" Jul 02 23:31:08.952880 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:31:08.952858 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426\": container with ID starting with c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426 not found: ID does not exist" containerID="c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426" Jul 02 23:31:08.952935 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.952888 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426"} err="failed to get container status \"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426\": rpc error: code = NotFound desc = could not find container \"c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426\": container with ID starting with c677edd792cd691e09c4d6b07af25a5eb5dede2891d02918c66eb8b292823426 not found: ID does not exist" Jul 02 23:31:08.955181 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955166 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxzd6\" (UniqueName: \"kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6\") pod \"517811fb-470a-4f6f-9a76-6da29e9a32ec\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " Jul 02 23:31:08.955268 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955196 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume\") pod \"517811fb-470a-4f6f-9a76-6da29e9a32ec\" (UID: \"517811fb-470a-4f6f-9a76-6da29e9a32ec\") " Jul 02 23:31:08.955364 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvk5\" (UniqueName: \"kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.955424 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.955636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955613 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "517811fb-470a-4f6f-9a76-6da29e9a32ec" (UID: "517811fb-470a-4f6f-9a76-6da29e9a32ec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:31:08.955704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.955671 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:08.957318 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.957265 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6" (OuterVolumeSpecName: "kube-api-access-zxzd6") pod "517811fb-470a-4f6f-9a76-6da29e9a32ec" (UID: "517811fb-470a-4f6f-9a76-6da29e9a32ec"). InnerVolumeSpecName "kube-api-access-zxzd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:31:08.962821 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:08.962805 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvk5\" (UniqueName: \"kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5\") pod \"kuadrant-operator-controller-manager-86878c769f-n8745\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:09.045197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.045154 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:09.056115 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.056088 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxzd6\" (UniqueName: \"kubernetes.io/projected/517811fb-470a-4f6f-9a76-6da29e9a32ec-kube-api-access-zxzd6\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:31:09.056115 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.056118 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/517811fb-470a-4f6f-9a76-6da29e9a32ec-extensions-socket-volume\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:31:09.176751 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.176728 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:31:09.178353 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:31:09.178328 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57c9ec8_ee49_471a_8db3_c0b7f315082a.slice/crio-7b2ff7174473e70aa03b2f03b03dec5cceb517c1c81c376f134a1257880b198f WatchSource:0}: Error finding container 7b2ff7174473e70aa03b2f03b03dec5cceb517c1c81c376f134a1257880b198f: Status 404 returned error can't find the container with id 7b2ff7174473e70aa03b2f03b03dec5cceb517c1c81c376f134a1257880b198f Jul 02 23:31:09.265034 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.265010 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:31:09.268504 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.268484 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lp9w4"] Jul 02 23:31:09.946162 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.946127 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" event={"ID":"a57c9ec8-ee49-471a-8db3-c0b7f315082a","Type":"ContainerStarted","Data":"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56"} Jul 02 23:31:09.946633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.946180 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" event={"ID":"a57c9ec8-ee49-471a-8db3-c0b7f315082a","Type":"ContainerStarted","Data":"7b2ff7174473e70aa03b2f03b03dec5cceb517c1c81c376f134a1257880b198f"} Jul 02 23:31:09.946633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.946262 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:09.966137 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:09.966085 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" podStartSLOduration=1.966067887 podStartE2EDuration="1.966067887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:31:09.964677915 +0000 UTC m=+524.047001842" watchObservedRunningTime="2026-07-02 23:31:09.966067887 +0000 UTC m=+524.048391810" Jul 02 23:31:10.444903 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:10.444868 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517811fb-470a-4f6f-9a76-6da29e9a32ec" path="/var/lib/kubelet/pods/517811fb-470a-4f6f-9a76-6da29e9a32ec/volumes" Jul 02 23:31:20.953725 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:20.953651 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:31:22.624631 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:22.624597 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:22.671450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:22.671418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:24.880939 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:24.880907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:25.236276 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:25.236178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:38.627693 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.627639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:38.628144 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.627904 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:38.628274 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.628260 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517811fb-470a-4f6f-9a76-6da29e9a32ec" containerName="manager" Jul 02 23:31:38.628352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.628279 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="517811fb-470a-4f6f-9a76-6da29e9a32ec" containerName="manager" Jul 02 23:31:38.628415 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.628404 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="517811fb-470a-4f6f-9a76-6da29e9a32ec" containerName="manager" Jul 02 23:31:38.630470 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.630456 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.632744 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.632719 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 23:31:38.632869 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.632723 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-svhmc\"" Jul 02 23:31:38.638505 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.638485 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:38.702625 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.702589 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqt9\" (UniqueName: \"kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.702972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.702936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.724348 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.724317 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:38.730560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.730533 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:38.779564 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.779531 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:38.804019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.803988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.804151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.804041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqt9\" (UniqueName: \"kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.804745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.804726 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.811962 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.811930 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqt9\" (UniqueName: \"kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9\") pod \"limitador-limitador-64c94fd64f-6vk8p\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:38.941599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:38.941508 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:39.081083 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:39.081055 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:39.083253 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:31:39.083227 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48032878_a6b1_4e88_9b8d_76676f0684a9.slice/crio-bf3d477fd6cb948799b6710c57c1bca9111529c492b8f3ca27ed6e992509bb2c WatchSource:0}: Error finding container bf3d477fd6cb948799b6710c57c1bca9111529c492b8f3ca27ed6e992509bb2c: Status 404 returned error can't find the container with id bf3d477fd6cb948799b6710c57c1bca9111529c492b8f3ca27ed6e992509bb2c Jul 02 23:31:39.097816 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:39.097794 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" event={"ID":"48032878-a6b1-4e88-9b8d-76676f0684a9","Type":"ContainerStarted","Data":"bf3d477fd6cb948799b6710c57c1bca9111529c492b8f3ca27ed6e992509bb2c"} Jul 02 23:31:39.443054 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:39.443017 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:39.505090 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:39.505057 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:42.118915 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:42.118882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" event={"ID":"48032878-a6b1-4e88-9b8d-76676f0684a9","Type":"ContainerStarted","Data":"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a"} Jul 02 23:31:42.119356 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:42.118961 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:42.136617 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:42.136572 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" podStartSLOduration=1.598171207 podStartE2EDuration="4.136557562s" totalImagesPullingTime="2.538386355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:31:42.134685912 +0000 UTC m=+556.217009836" watchObservedRunningTime="2026-07-02 23:31:42.136557562 +0000 UTC m=+556.218881486" Jul 02 23:31:53.125300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:53.125247 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:55.355056 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.355021 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:55.355473 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.355216 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" podUID="48032878-a6b1-4e88-9b8d-76676f0684a9" containerName="limitador" containerID="cri-o://027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a" gracePeriod=30 Jul 02 23:31:55.904634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.904610 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:55.950001 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.949914 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrqt9\" (UniqueName: \"kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9\") pod \"48032878-a6b1-4e88-9b8d-76676f0684a9\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " Jul 02 23:31:55.950001 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.950000 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file\") pod \"48032878-a6b1-4e88-9b8d-76676f0684a9\" (UID: \"48032878-a6b1-4e88-9b8d-76676f0684a9\") " Jul 02 23:31:55.950330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.950275 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file" (OuterVolumeSpecName: "config-file") pod "48032878-a6b1-4e88-9b8d-76676f0684a9" (UID: "48032878-a6b1-4e88-9b8d-76676f0684a9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:31:55.952110 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:55.952078 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9" (OuterVolumeSpecName: "kube-api-access-lrqt9") pod "48032878-a6b1-4e88-9b8d-76676f0684a9" (UID: "48032878-a6b1-4e88-9b8d-76676f0684a9"). InnerVolumeSpecName "kube-api-access-lrqt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:31:56.051173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.051145 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48032878-a6b1-4e88-9b8d-76676f0684a9-config-file\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:31:56.051173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.051168 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqt9\" (UniqueName: \"kubernetes.io/projected/48032878-a6b1-4e88-9b8d-76676f0684a9-kube-api-access-lrqt9\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:31:56.201810 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.201713 2561 generic.go:356] "Generic (PLEG): container finished" podID="48032878-a6b1-4e88-9b8d-76676f0684a9" containerID="027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a" exitCode=0 Jul 02 23:31:56.201810 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.201760 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" Jul 02 23:31:56.201810 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.201779 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" event={"ID":"48032878-a6b1-4e88-9b8d-76676f0684a9","Type":"ContainerDied","Data":"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a"} Jul 02 23:31:56.202091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.201821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6vk8p" event={"ID":"48032878-a6b1-4e88-9b8d-76676f0684a9","Type":"ContainerDied","Data":"bf3d477fd6cb948799b6710c57c1bca9111529c492b8f3ca27ed6e992509bb2c"} Jul 02 23:31:56.202091 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.201843 2561 scope.go:117] "RemoveContainer" containerID="027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a" Jul 02 23:31:56.211140 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.211122 2561 scope.go:117] "RemoveContainer" containerID="027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a" Jul 02 23:31:56.211414 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:31:56.211396 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a\": container with ID starting with 027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a not found: ID does not exist" containerID="027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a" Jul 02 23:31:56.211488 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.211424 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a"} err="failed to get container status \"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a\": rpc error: code = NotFound desc = could not find container \"027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a\": container with ID starting with 027075f8ee637532063c2f09f9e4c6e30bec08f94cc5b92f94644c69df06cd6a not found: ID does not exist" Jul 02 23:31:56.229134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.228440 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:56.229134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.228487 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6vk8p"] Jul 02 23:31:56.445100 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:56.445066 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48032878-a6b1-4e88-9b8d-76676f0684a9" path="/var/lib/kubelet/pods/48032878-a6b1-4e88-9b8d-76676f0684a9/volumes" Jul 02 23:31:58.467107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.467075 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-46c9c"] Jul 02 23:31:58.467619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.467472 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48032878-a6b1-4e88-9b8d-76676f0684a9" containerName="limitador" Jul 02 23:31:58.467619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.467486 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48032878-a6b1-4e88-9b8d-76676f0684a9" containerName="limitador" Jul 02 23:31:58.467619 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.467604 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48032878-a6b1-4e88-9b8d-76676f0684a9" containerName="limitador" Jul 02 23:31:58.471920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.471898 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.474712 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.474612 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 23:31:58.475500 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.475441 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wcjcc\"" Jul 02 23:31:58.478114 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.478088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-46c9c"] Jul 02 23:31:58.498792 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.498767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:31:58.573008 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.572970 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9ad423f-a519-44aa-8efb-90110de60c6d-data\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.573175 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.573015 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9rl\" (UniqueName: \"kubernetes.io/projected/d9ad423f-a519-44aa-8efb-90110de60c6d-kube-api-access-tk9rl\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.674433 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.674398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9ad423f-a519-44aa-8efb-90110de60c6d-data\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.674628 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.674441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9rl\" (UniqueName: \"kubernetes.io/projected/d9ad423f-a519-44aa-8efb-90110de60c6d-kube-api-access-tk9rl\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.674764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.674742 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9ad423f-a519-44aa-8efb-90110de60c6d-data\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.681985 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.681960 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9rl\" (UniqueName: \"kubernetes.io/projected/d9ad423f-a519-44aa-8efb-90110de60c6d-kube-api-access-tk9rl\") pod \"postgres-69fb4594cf-46c9c\" (UID: \"d9ad423f-a519-44aa-8efb-90110de60c6d\") " pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.784937 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.784905 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:31:58.917519 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:58.917489 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-46c9c"] Jul 02 23:31:58.919568 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:31:58.919544 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ad423f_a519_44aa_8efb_90110de60c6d.slice/crio-2c6a6f2ee0e04d4b9fa480eafcd367de2c17165b1d93b3e7f978edd23da3ca7b WatchSource:0}: Error finding container 2c6a6f2ee0e04d4b9fa480eafcd367de2c17165b1d93b3e7f978edd23da3ca7b: Status 404 returned error can't find the container with id 2c6a6f2ee0e04d4b9fa480eafcd367de2c17165b1d93b3e7f978edd23da3ca7b Jul 02 23:31:59.220256 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:31:59.220173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-46c9c" event={"ID":"d9ad423f-a519-44aa-8efb-90110de60c6d","Type":"ContainerStarted","Data":"2c6a6f2ee0e04d4b9fa480eafcd367de2c17165b1d93b3e7f978edd23da3ca7b"} Jul 02 23:32:04.259197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:04.259112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-46c9c" event={"ID":"d9ad423f-a519-44aa-8efb-90110de60c6d","Type":"ContainerStarted","Data":"9fb5eabe57219702f794dba9a6b9d68f9bbb8f899024922c398fae95c6580fe2"} Jul 02 23:32:04.259197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:04.259181 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:32:04.274019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:04.273972 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-46c9c" podStartSLOduration=1.214352423 podStartE2EDuration="6.273957407s" totalImagesPullingTime="5.059604984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:32:04.272861991 +0000 UTC m=+578.355185915" watchObservedRunningTime="2026-07-02 23:32:04.273957407 +0000 UTC m=+578.356281332" Jul 02 23:32:10.295851 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:10.295823 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-46c9c" Jul 02 23:32:10.603506 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:10.603430 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:10.839363 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:10.839336 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:17.083807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.083774 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:17.095334 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.095301 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:17.115900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.115869 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:17.116066 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.115987 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.118677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.118654 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zkv95\"" Jul 02 23:32:17.118851 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.118683 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 23:32:17.118851 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.118716 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 23:32:17.177468 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.177432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:17.241524 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.241494 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw2d\" (UniqueName: \"kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.241684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.241554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.271445 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.271417 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:17.276211 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.276186 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:32:17.278771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.278749 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.287819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.287791 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:32:17.322455 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.322425 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:17.342107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.342025 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw2d\" (UniqueName: \"kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.342107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.342072 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.344675 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.344652 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.349435 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.349411 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw2d\" (UniqueName: \"kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d\") pod \"maas-controller-5f74c74f46-r8wk7\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.427234 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.427205 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:17.442658 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.442626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.442779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.442676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkv4\" (UniqueName: \"kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.543639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.543602 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.543790 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.543656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkv4\" (UniqueName: \"kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.546055 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.546032 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.553651 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.553630 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkv4\" (UniqueName: \"kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4\") pod \"maas-controller-cddbb748b-xm62x\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.560479 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.560454 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:17.562103 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:32:17.562078 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c79aea3_7518_4234_bcb4_90cba24d5e7e.slice/crio-0a90acbed81b310b16a96c5fe3b077aa9a7cb2638de73b3de9bcb41d73a14a23 WatchSource:0}: Error finding container 0a90acbed81b310b16a96c5fe3b077aa9a7cb2638de73b3de9bcb41d73a14a23: Status 404 returned error can't find the container with id 0a90acbed81b310b16a96c5fe3b077aa9a7cb2638de73b3de9bcb41d73a14a23 Jul 02 23:32:17.590612 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.590586 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:17.721191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:17.721166 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:32:17.722322 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:32:17.722277 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a053d15_d630_42ef_916b_15e55c8f40cf.slice/crio-564ea54bc14552439bdc7a2bd2bc66d94326ad8efc26babba8affe6886eb6615 WatchSource:0}: Error finding container 564ea54bc14552439bdc7a2bd2bc66d94326ad8efc26babba8affe6886eb6615: Status 404 returned error can't find the container with id 564ea54bc14552439bdc7a2bd2bc66d94326ad8efc26babba8affe6886eb6615 Jul 02 23:32:18.339790 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:18.339684 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" event={"ID":"8c79aea3-7518-4234-bcb4-90cba24d5e7e","Type":"ContainerStarted","Data":"0a90acbed81b310b16a96c5fe3b077aa9a7cb2638de73b3de9bcb41d73a14a23"} Jul 02 23:32:18.341975 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:18.341893 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-xm62x" event={"ID":"9a053d15-d630-42ef-916b-15e55c8f40cf","Type":"ContainerStarted","Data":"564ea54bc14552439bdc7a2bd2bc66d94326ad8efc26babba8affe6886eb6615"} Jul 02 23:32:19.868476 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:19.868440 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:19.964821 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:19.964745 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:20.354623 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.354588 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" event={"ID":"8c79aea3-7518-4234-bcb4-90cba24d5e7e","Type":"ContainerStarted","Data":"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b"} Jul 02 23:32:20.354832 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.354654 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:20.356234 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.356208 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-xm62x" event={"ID":"9a053d15-d630-42ef-916b-15e55c8f40cf","Type":"ContainerStarted","Data":"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5"} Jul 02 23:32:20.356374 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.356263 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:20.376142 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.376078 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" podStartSLOduration=1.304299697 podStartE2EDuration="3.376060541s" totalImagesPullingTime="2.071760844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:32:20.373701326 +0000 UTC m=+594.456025249" watchObservedRunningTime="2026-07-02 23:32:20.376060541 +0000 UTC m=+594.458384466" Jul 02 23:32:20.397302 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:20.394526 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cddbb748b-xm62x" podStartSLOduration=1.489637571 podStartE2EDuration="3.394508324s" totalImagesPullingTime="1.904870753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:32:20.393004151 +0000 UTC m=+594.475328076" watchObservedRunningTime="2026-07-02 23:32:20.394508324 +0000 UTC m=+594.476832250" Jul 02 23:32:26.343719 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:26.343687 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:32:26.344951 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:26.344931 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:32:31.366252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.366216 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:32:31.366852 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.366325 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:31.425706 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.425670 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:31.425952 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.425922 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" podUID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" containerName="manager" containerID="cri-o://b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b" gracePeriod=30 Jul 02 23:32:31.673171 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.673147 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:31.769014 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.768986 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert\") pod \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " Jul 02 23:32:31.769163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.769024 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrw2d\" (UniqueName: \"kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d\") pod \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\" (UID: \"8c79aea3-7518-4234-bcb4-90cba24d5e7e\") " Jul 02 23:32:31.771386 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.771356 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d" (OuterVolumeSpecName: "kube-api-access-zrw2d") pod "8c79aea3-7518-4234-bcb4-90cba24d5e7e" (UID: "8c79aea3-7518-4234-bcb4-90cba24d5e7e"). InnerVolumeSpecName "kube-api-access-zrw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:32:31.771386 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.771362 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8c79aea3-7518-4234-bcb4-90cba24d5e7e" (UID: "8c79aea3-7518-4234-bcb4-90cba24d5e7e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:32:31.870013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.869978 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c79aea3-7518-4234-bcb4-90cba24d5e7e-webhook-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:32:31.870013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:31.870009 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrw2d\" (UniqueName: \"kubernetes.io/projected/8c79aea3-7518-4234-bcb4-90cba24d5e7e-kube-api-access-zrw2d\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:32:32.427819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.427708 2561 generic.go:356] "Generic (PLEG): container finished" podID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" containerID="b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b" exitCode=0 Jul 02 23:32:32.427819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.427752 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" Jul 02 23:32:32.427819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.427798 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" event={"ID":"8c79aea3-7518-4234-bcb4-90cba24d5e7e","Type":"ContainerDied","Data":"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b"} Jul 02 23:32:32.439842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.427849 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f74c74f46-r8wk7" event={"ID":"8c79aea3-7518-4234-bcb4-90cba24d5e7e","Type":"ContainerDied","Data":"0a90acbed81b310b16a96c5fe3b077aa9a7cb2638de73b3de9bcb41d73a14a23"} Jul 02 23:32:32.439842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.427874 2561 scope.go:117] "RemoveContainer" containerID="b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b" Jul 02 23:32:32.439842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.437151 2561 scope.go:117] "RemoveContainer" containerID="b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b" Jul 02 23:32:32.439842 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:32:32.437413 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b\": container with ID starting with b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b not found: ID does not exist" containerID="b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b" Jul 02 23:32:32.439842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.437439 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b"} err="failed to get container status \"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b\": rpc error: code = NotFound desc = could not find container \"b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b\": container with ID starting with b6ab4085ebeadf92bded492643e4628e0b4f7786dc30d9f551584fd7577d168b not found: ID does not exist" Jul 02 23:32:32.449155 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.449123 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:32.452793 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:32.452774 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f74c74f46-r8wk7"] Jul 02 23:32:34.444375 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:34.444334 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" path="/var/lib/kubelet/pods/8c79aea3-7518-4234-bcb4-90cba24d5e7e/volumes" Jul 02 23:32:52.441032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.440937 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:52.927341 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.927306 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-758bb66cf4-78d6f"] Jul 02 23:32:52.927736 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.927713 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" containerName="manager" Jul 02 23:32:52.927736 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.927732 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" containerName="manager" Jul 02 23:32:52.927911 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.927811 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c79aea3-7518-4234-bcb4-90cba24d5e7e" containerName="manager" Jul 02 23:32:52.931009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.930990 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:52.933582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.933558 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 02 23:32:52.933582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.933574 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-mfwkn\"" Jul 02 23:32:52.945758 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:52.945735 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-758bb66cf4-78d6f"] Jul 02 23:32:53.060947 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.060915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21ee108-a2df-4902-8ac4-60d41eac1a0a-openshift-service-ca\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.061148 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.060955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm4d\" (UniqueName: \"kubernetes.io/projected/c21ee108-a2df-4902-8ac4-60d41eac1a0a-kube-api-access-flm4d\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.061148 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.061079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c21ee108-a2df-4902-8ac4-60d41eac1a0a-maas-api-tls\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.161751 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.161711 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c21ee108-a2df-4902-8ac4-60d41eac1a0a-maas-api-tls\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.161940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.161777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21ee108-a2df-4902-8ac4-60d41eac1a0a-openshift-service-ca\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.161940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.161814 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm4d\" (UniqueName: \"kubernetes.io/projected/c21ee108-a2df-4902-8ac4-60d41eac1a0a-kube-api-access-flm4d\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.162536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.162469 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21ee108-a2df-4902-8ac4-60d41eac1a0a-openshift-service-ca\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.164419 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.164398 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c21ee108-a2df-4902-8ac4-60d41eac1a0a-maas-api-tls\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.172978 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.172957 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm4d\" (UniqueName: \"kubernetes.io/projected/c21ee108-a2df-4902-8ac4-60d41eac1a0a-kube-api-access-flm4d\") pod \"maas-api-758bb66cf4-78d6f\" (UID: \"c21ee108-a2df-4902-8ac4-60d41eac1a0a\") " pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.241740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.241645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:53.393434 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.393246 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-tzvtn"] Jul 02 23:32:53.398539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.398513 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-758bb66cf4-78d6f"] Jul 02 23:32:53.398539 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:32:53.398533 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21ee108_a2df_4902_8ac4_60d41eac1a0a.slice/crio-546c2c8a26809c322fd101b4a5d9fae6277f60aba52c0ec8389cf2fc335517f0 WatchSource:0}: Error finding container 546c2c8a26809c322fd101b4a5d9fae6277f60aba52c0ec8389cf2fc335517f0: Status 404 returned error can't find the container with id 546c2c8a26809c322fd101b4a5d9fae6277f60aba52c0ec8389cf2fc335517f0 Jul 02 23:32:53.398716 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.398588 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.399945 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.399927 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:32:53.402334 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.402313 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 23:32:53.402443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.402335 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lkhx2\"" Jul 02 23:32:53.409985 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.409960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-tzvtn"] Jul 02 23:32:53.524805 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.524769 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:32:53.545446 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.545409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-758bb66cf4-78d6f" event={"ID":"c21ee108-a2df-4902-8ac4-60d41eac1a0a","Type":"ContainerStarted","Data":"546c2c8a26809c322fd101b4a5d9fae6277f60aba52c0ec8389cf2fc335517f0"} Jul 02 23:32:53.565899 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.565864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-plugins-config-volume\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.566073 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.565915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xcpg\" (UniqueName: \"kubernetes.io/projected/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-kube-api-access-4xcpg\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.667144 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.667113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-plugins-config-volume\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.667370 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.667153 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcpg\" (UniqueName: \"kubernetes.io/projected/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-kube-api-access-4xcpg\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.667910 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.667885 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-plugins-config-volume\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.674895 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.674874 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcpg\" (UniqueName: \"kubernetes.io/projected/c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff-kube-api-access-4xcpg\") pod \"payload-pre-processing-7964457696-tzvtn\" (UID: \"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff\") " pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.717548 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.717519 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:53.855155 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:53.855128 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-tzvtn"] Jul 02 23:32:53.857004 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:32:53.856978 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cf40d6_c9a6_40dc_a9bc_ddbbd66641ff.slice/crio-c974baac2f6c62642865bb27d10e20db44342cc0b8ff18977f71ce51cf1602ee WatchSource:0}: Error finding container c974baac2f6c62642865bb27d10e20db44342cc0b8ff18977f71ce51cf1602ee: Status 404 returned error can't find the container with id c974baac2f6c62642865bb27d10e20db44342cc0b8ff18977f71ce51cf1602ee Jul 02 23:32:54.553514 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:54.553451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" event={"ID":"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff","Type":"ContainerStarted","Data":"c974baac2f6c62642865bb27d10e20db44342cc0b8ff18977f71ce51cf1602ee"} Jul 02 23:32:56.567344 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.567312 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-758bb66cf4-78d6f" event={"ID":"c21ee108-a2df-4902-8ac4-60d41eac1a0a","Type":"ContainerStarted","Data":"8b3e2b22ebe59385f836e18d4587d4a21c19b19c80a972ce60787173ccf6438c"} Jul 02 23:32:56.567807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.567435 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:32:56.568895 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.568876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" event={"ID":"c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff","Type":"ContainerStarted","Data":"84930294fc950c919fb878bd012ec633adbc5538567ba94fdafdf3826af62999"} Jul 02 23:32:56.568990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.568913 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:32:56.584366 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.584325 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-758bb66cf4-78d6f" podStartSLOduration=2.454544902 podStartE2EDuration="4.584311301s" totalImagesPullingTime="2.129766399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:32:56.581989564 +0000 UTC m=+630.664313491" watchObservedRunningTime="2026-07-02 23:32:56.584311301 +0000 UTC m=+630.666635225" Jul 02 23:32:56.600366 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:32:56.600320 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" podStartSLOduration=1.932786433 podStartE2EDuration="3.600306375s" totalImagesPullingTime="1.667519942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:32:56.598998189 +0000 UTC m=+630.681322113" watchObservedRunningTime="2026-07-02 23:32:56.600306375 +0000 UTC m=+630.682630298" Jul 02 23:33:02.580488 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:02.580458 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-758bb66cf4-78d6f" Jul 02 23:33:03.555153 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:03.555089 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:03.647041 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:03.647009 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:07.576354 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:07.576324 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-tzvtn" Jul 02 23:33:16.753369 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:16.753332 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:19.924319 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:19.924257 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:20.025681 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:20.025653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:20.889068 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:20.889039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:26.467070 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.466327 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pshls"] Jul 02 23:33:26.472360 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.472332 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.475550 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.475523 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k5pld\"" Jul 02 23:33:26.475775 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.475755 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 02 23:33:26.475858 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.475801 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 23:33:26.475915 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.475865 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 23:33:26.478921 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.478850 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pshls"] Jul 02 23:33:26.489454 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.489435 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:26.550352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550320 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.550352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2w55\" (UniqueName: \"kubernetes.io/projected/217b86b7-f271-4643-8cdb-b883dafc66a1-kube-api-access-z2w55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.550536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.550536 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217b86b7-f271-4643-8cdb-b883dafc66a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.550608 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550566 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.550642 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.550629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651378 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651378 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217b86b7-f271-4643-8cdb-b883dafc66a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651378 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651493 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2w55\" (UniqueName: \"kubernetes.io/projected/217b86b7-f271-4643-8cdb-b883dafc66a1-kube-api-access-z2w55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651811 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651722 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651811 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651725 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.651935 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.651860 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.653848 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.653824 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217b86b7-f271-4643-8cdb-b883dafc66a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.654062 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.654042 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217b86b7-f271-4643-8cdb-b883dafc66a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.658990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.658955 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2w55\" (UniqueName: \"kubernetes.io/projected/217b86b7-f271-4643-8cdb-b883dafc66a1-kube-api-access-z2w55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pshls\" (UID: \"217b86b7-f271-4643-8cdb-b883dafc66a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.785878 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.785843 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:26.927328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:26.927225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pshls"] Jul 02 23:33:26.929718 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:33:26.929694 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217b86b7_f271_4643_8cdb_b883dafc66a1.slice/crio-c87fb42945b49378adc24b0142f7d1ab3c6c8b015ff773d29f1c85c5e571c6ea WatchSource:0}: Error finding container c87fb42945b49378adc24b0142f7d1ab3c6c8b015ff773d29f1c85c5e571c6ea: Status 404 returned error can't find the container with id c87fb42945b49378adc24b0142f7d1ab3c6c8b015ff773d29f1c85c5e571c6ea Jul 02 23:33:27.743003 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:27.742968 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" event={"ID":"217b86b7-f271-4643-8cdb-b883dafc66a1","Type":"ContainerStarted","Data":"c87fb42945b49378adc24b0142f7d1ab3c6c8b015ff773d29f1c85c5e571c6ea"} Jul 02 23:33:28.753454 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:28.753416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" event={"ID":"217b86b7-f271-4643-8cdb-b883dafc66a1","Type":"ContainerStarted","Data":"370f4000af25f2c5d5da3ea760ad67024225cd8f38154a9e577f8badfc60adec"} Jul 02 23:33:28.768812 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:28.768710 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" podStartSLOduration=1.747581509 podStartE2EDuration="2.76869174s" totalImagesPullingTime="1.021110231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:33:28.768105341 +0000 UTC m=+662.850429266" watchObservedRunningTime="2026-07-02 23:33:28.76869174 +0000 UTC m=+662.851015667" Jul 02 23:33:32.801127 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:32.801092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:36.786994 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:36.786956 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:36.787498 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:36.787045 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:36.800015 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:36.799988 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:37.816665 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:37.816636 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pshls" Jul 02 23:33:47.381261 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.381223 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb"] Jul 02 23:33:47.386117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.386099 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.388707 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.388687 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 02 23:33:47.394482 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.394456 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb"] Jul 02 23:33:47.406628 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.406603 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:47.545618 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.545779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.545779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64abc7f-584c-4318-8953-8a22a57740a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.545779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24lt\" (UniqueName: \"kubernetes.io/projected/a64abc7f-584c-4318-8953-8a22a57740a9-kube-api-access-p24lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.545926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.545926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.545897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647025 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.646941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647025 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.646982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647025 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647139 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64abc7f-584c-4318-8953-8a22a57740a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647209 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24lt\" (UniqueName: \"kubernetes.io/projected/a64abc7f-584c-4318-8953-8a22a57740a9-kube-api-access-p24lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647392 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647416 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.647606 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.647520 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.649589 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.649569 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a64abc7f-584c-4318-8953-8a22a57740a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.650029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.650006 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64abc7f-584c-4318-8953-8a22a57740a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.654251 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.654229 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24lt\" (UniqueName: \"kubernetes.io/projected/a64abc7f-584c-4318-8953-8a22a57740a9-kube-api-access-p24lt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb\" (UID: \"a64abc7f-584c-4318-8953-8a22a57740a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.697358 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.697332 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:47.832496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.832466 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb"] Jul 02 23:33:47.833844 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:33:47.833808 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64abc7f_584c_4318_8953_8a22a57740a9.slice/crio-b7055f0842fa869e754375de5f1e0c9392605194dcf027e1cf7d83ba3edcbd45 WatchSource:0}: Error finding container b7055f0842fa869e754375de5f1e0c9392605194dcf027e1cf7d83ba3edcbd45: Status 404 returned error can't find the container with id b7055f0842fa869e754375de5f1e0c9392605194dcf027e1cf7d83ba3edcbd45 Jul 02 23:33:47.861367 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:47.861336 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" event={"ID":"a64abc7f-584c-4318-8953-8a22a57740a9","Type":"ContainerStarted","Data":"b7055f0842fa869e754375de5f1e0c9392605194dcf027e1cf7d83ba3edcbd45"} Jul 02 23:33:48.868807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:48.868776 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" event={"ID":"a64abc7f-584c-4318-8953-8a22a57740a9","Type":"ContainerStarted","Data":"6068836c1766b132968319cb4c8578bf03c6b0798cfec863bbdfda04a638018f"} Jul 02 23:33:48.887958 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:48.887901 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" podStartSLOduration=1.7218657 podStartE2EDuration="1.887880379s" totalImagesPullingTime="166.014679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:33:48.885935393 +0000 UTC m=+682.968259340" watchObservedRunningTime="2026-07-02 23:33:48.887880379 +0000 UTC m=+682.970204304" Jul 02 23:33:53.290472 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:53.290438 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:33:57.698036 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:57.698001 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:57.698491 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:57.698063 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:57.710881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:57.710851 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:33:57.937988 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:33:57.937962 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb" Jul 02 23:34:12.091126 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:34:12.091082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:13.373451 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:35:13.373408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:13.421246 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:35:13.421214 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:23.901646 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:35:23.901614 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.345750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:36:00.345664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.444735 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:36:00.444705 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.504677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:36:01.504637 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.599878 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:36:01.599841 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:12.257076 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.257041 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:12.371229 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.371195 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:37:12.371448 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.371423 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-cddbb748b-xm62x" podUID="9a053d15-d630-42ef-916b-15e55c8f40cf" containerName="manager" containerID="cri-o://d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5" gracePeriod=30 Jul 02 23:37:12.633278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.633253 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:37:12.695816 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.695782 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgkv4\" (UniqueName: \"kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4\") pod \"9a053d15-d630-42ef-916b-15e55c8f40cf\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " Jul 02 23:37:12.696018 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.695893 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert\") pod \"9a053d15-d630-42ef-916b-15e55c8f40cf\" (UID: \"9a053d15-d630-42ef-916b-15e55c8f40cf\") " Jul 02 23:37:12.698004 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.697973 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4" (OuterVolumeSpecName: "kube-api-access-lgkv4") pod "9a053d15-d630-42ef-916b-15e55c8f40cf" (UID: "9a053d15-d630-42ef-916b-15e55c8f40cf"). InnerVolumeSpecName "kube-api-access-lgkv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:37:12.698228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.698214 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9a053d15-d630-42ef-916b-15e55c8f40cf" (UID: "9a053d15-d630-42ef-916b-15e55c8f40cf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:37:12.796677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.796599 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a053d15-d630-42ef-916b-15e55c8f40cf-webhook-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:37:12.796677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.796625 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkv4\" (UniqueName: \"kubernetes.io/projected/9a053d15-d630-42ef-916b-15e55c8f40cf-kube-api-access-lgkv4\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:37:12.986012 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.985968 2561 generic.go:356] "Generic (PLEG): container finished" podID="9a053d15-d630-42ef-916b-15e55c8f40cf" containerID="d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5" exitCode=0 Jul 02 23:37:12.986012 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.986009 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-xm62x" Jul 02 23:37:12.986245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.986065 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-xm62x" event={"ID":"9a053d15-d630-42ef-916b-15e55c8f40cf","Type":"ContainerDied","Data":"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5"} Jul 02 23:37:12.986245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.986129 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-xm62x" event={"ID":"9a053d15-d630-42ef-916b-15e55c8f40cf","Type":"ContainerDied","Data":"564ea54bc14552439bdc7a2bd2bc66d94326ad8efc26babba8affe6886eb6615"} Jul 02 23:37:12.986245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.986159 2561 scope.go:117] "RemoveContainer" containerID="d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5" Jul 02 23:37:12.995855 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.995838 2561 scope.go:117] "RemoveContainer" containerID="d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5" Jul 02 23:37:12.996135 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:37:12.996108 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5\": container with ID starting with d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5 not found: ID does not exist" containerID="d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5" Jul 02 23:37:12.996186 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:12.996144 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5"} err="failed to get container status \"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5\": rpc error: code = NotFound desc = could not find container \"d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5\": container with ID starting with d7cc5fe0195a3dbf2d59743f50c1296a5c2bd5f64790a7adbcc94efccb0b91f5 not found: ID does not exist" Jul 02 23:37:13.007468 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.007439 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:37:13.010550 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.010525 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cddbb748b-xm62x"] Jul 02 23:37:13.660735 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.660656 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:13.774541 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.774513 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cddbb748b-hpcrh"] Jul 02 23:37:13.774922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.774910 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a053d15-d630-42ef-916b-15e55c8f40cf" containerName="manager" Jul 02 23:37:13.774974 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.774923 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a053d15-d630-42ef-916b-15e55c8f40cf" containerName="manager" Jul 02 23:37:13.775009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.775003 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a053d15-d630-42ef-916b-15e55c8f40cf" containerName="manager" Jul 02 23:37:13.779294 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.779270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.783339 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.783314 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zkv95\"" Jul 02 23:37:13.783884 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.783323 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 23:37:13.784380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.783458 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 23:37:13.787450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.787140 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cddbb748b-hpcrh"] Jul 02 23:37:13.803026 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.802997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6179bbdb-6402-40a9-b296-68edeb54ef69-webhook-cert\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.803133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.803056 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9l4\" (UniqueName: \"kubernetes.io/projected/6179bbdb-6402-40a9-b296-68edeb54ef69-kube-api-access-7n9l4\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.903714 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.903674 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6179bbdb-6402-40a9-b296-68edeb54ef69-webhook-cert\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.903900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.903740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9l4\" (UniqueName: \"kubernetes.io/projected/6179bbdb-6402-40a9-b296-68edeb54ef69-kube-api-access-7n9l4\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.906278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.906256 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6179bbdb-6402-40a9-b296-68edeb54ef69-webhook-cert\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:13.911559 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:13.911503 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9l4\" (UniqueName: \"kubernetes.io/projected/6179bbdb-6402-40a9-b296-68edeb54ef69-kube-api-access-7n9l4\") pod \"maas-controller-cddbb748b-hpcrh\" (UID: \"6179bbdb-6402-40a9-b296-68edeb54ef69\") " pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:14.101203 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:14.101163 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:14.243841 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:14.243815 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cddbb748b-hpcrh"] Jul 02 23:37:14.245001 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:37:14.244974 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6179bbdb_6402_40a9_b296_68edeb54ef69.slice/crio-106a4c97dddf0e715532c933e6fcb0c9950a924f4c9083261063ede0dedd519f WatchSource:0}: Error finding container 106a4c97dddf0e715532c933e6fcb0c9950a924f4c9083261063ede0dedd519f: Status 404 returned error can't find the container with id 106a4c97dddf0e715532c933e6fcb0c9950a924f4c9083261063ede0dedd519f Jul 02 23:37:14.445607 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:14.445534 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a053d15-d630-42ef-916b-15e55c8f40cf" path="/var/lib/kubelet/pods/9a053d15-d630-42ef-916b-15e55c8f40cf/volumes" Jul 02 23:37:15.001117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:15.001080 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-hpcrh" event={"ID":"6179bbdb-6402-40a9-b296-68edeb54ef69","Type":"ContainerStarted","Data":"0b4ed9f07018ad2359d65a0e6dae2386e85ee9b336a71a10a3d2e170b3bb64c8"} Jul 02 23:37:15.001517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:15.001133 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cddbb748b-hpcrh" event={"ID":"6179bbdb-6402-40a9-b296-68edeb54ef69","Type":"ContainerStarted","Data":"106a4c97dddf0e715532c933e6fcb0c9950a924f4c9083261063ede0dedd519f"} Jul 02 23:37:15.001517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:15.001158 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:15.017758 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:15.017709 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cddbb748b-hpcrh" podStartSLOduration=1.473920512 podStartE2EDuration="2.017691966s" totalImagesPullingTime="543.771454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:37:15.015160668 +0000 UTC m=+889.097484593" watchObservedRunningTime="2026-07-02 23:37:15.017691966 +0000 UTC m=+889.100015891" Jul 02 23:37:26.013081 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:26.013006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cddbb748b-hpcrh" Jul 02 23:37:26.381877 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:26.381790 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:37:26.384684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:26.384662 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:37:49.111693 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:49.111657 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:49.209455 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:49.209414 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:49.322315 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:49.322256 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:55.075932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:55.075901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:56.201267 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:37:56.201233 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:01.234372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:38:01.234331 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:01.334330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:38:01.334271 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:50.474731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:39:50.474698 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:50.559918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:39:50.559881 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:08.551011 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:08.550977 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:54.253497 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:54.253414 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:41:54.254044 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:54.253615 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" podUID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" containerName="manager" containerID="cri-o://17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56" gracePeriod=10 Jul 02 23:41:57.211080 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.211057 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:41:57.307153 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.307112 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbvk5\" (UniqueName: \"kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5\") pod \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " Jul 02 23:41:57.307384 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.307256 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume\") pod \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\" (UID: \"a57c9ec8-ee49-471a-8db3-c0b7f315082a\") " Jul 02 23:41:57.307654 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.307632 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a57c9ec8-ee49-471a-8db3-c0b7f315082a" (UID: "a57c9ec8-ee49-471a-8db3-c0b7f315082a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:41:57.309324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.309300 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5" (OuterVolumeSpecName: "kube-api-access-pbvk5") pod "a57c9ec8-ee49-471a-8db3-c0b7f315082a" (UID: "a57c9ec8-ee49-471a-8db3-c0b7f315082a"). InnerVolumeSpecName "kube-api-access-pbvk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:41:57.408476 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.408432 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a57c9ec8-ee49-471a-8db3-c0b7f315082a-extensions-socket-volume\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:41:57.408476 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.408469 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvk5\" (UniqueName: \"kubernetes.io/projected/a57c9ec8-ee49-471a-8db3-c0b7f315082a-kube-api-access-pbvk5\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:41:57.595894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.595803 2561 generic.go:356] "Generic (PLEG): container finished" podID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" containerID="17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56" exitCode=0 Jul 02 23:41:57.595894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.595849 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" Jul 02 23:41:57.595894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.595863 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" event={"ID":"a57c9ec8-ee49-471a-8db3-c0b7f315082a","Type":"ContainerDied","Data":"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56"} Jul 02 23:41:57.596154 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.595907 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745" event={"ID":"a57c9ec8-ee49-471a-8db3-c0b7f315082a","Type":"ContainerDied","Data":"7b2ff7174473e70aa03b2f03b03dec5cceb517c1c81c376f134a1257880b198f"} Jul 02 23:41:57.596154 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.595929 2561 scope.go:117] "RemoveContainer" containerID="17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56" Jul 02 23:41:57.606790 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.606769 2561 scope.go:117] "RemoveContainer" containerID="17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56" Jul 02 23:41:57.607086 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:41:57.607065 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56\": container with ID starting with 17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56 not found: ID does not exist" containerID="17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56" Jul 02 23:41:57.607148 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.607095 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56"} err="failed to get container status \"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56\": rpc error: code = NotFound desc = could not find container \"17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56\": container with ID starting with 17835bfa62e19de5b198d9db7fb7a3576660401a07a306a29f4c109af891ff56 not found: ID does not exist" Jul 02 23:41:57.618612 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.618588 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:41:57.626643 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:57.626615 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n8745"] Jul 02 23:41:58.445247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:41:58.445204 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" path="/var/lib/kubelet/pods/a57c9ec8-ee49-471a-8db3-c0b7f315082a/volumes" Jul 02 23:42:26.416949 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:26.416921 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:42:26.421479 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:26.421455 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:42:41.440445 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:41.440407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:55.397123 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.397088 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw"] Jul 02 23:42:55.399518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.397496 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" containerName="manager" Jul 02 23:42:55.399518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.397508 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" containerName="manager" Jul 02 23:42:55.399518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.397582 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a57c9ec8-ee49-471a-8db3-c0b7f315082a" containerName="manager" Jul 02 23:42:55.400517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.400502 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.402940 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.402917 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ml5c8\"" Jul 02 23:42:55.416318 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.416277 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw"] Jul 02 23:42:55.532370 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.532339 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtztx\" (UniqueName: \"kubernetes.io/projected/da288574-77eb-4bd1-bf0c-9a083b54b2ba-kube-api-access-vtztx\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.532539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.532453 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da288574-77eb-4bd1-bf0c-9a083b54b2ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.633538 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.633512 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da288574-77eb-4bd1-bf0c-9a083b54b2ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.633711 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.633549 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtztx\" (UniqueName: \"kubernetes.io/projected/da288574-77eb-4bd1-bf0c-9a083b54b2ba-kube-api-access-vtztx\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.633861 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.633843 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da288574-77eb-4bd1-bf0c-9a083b54b2ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.641548 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.641527 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtztx\" (UniqueName: \"kubernetes.io/projected/da288574-77eb-4bd1-bf0c-9a083b54b2ba-kube-api-access-vtztx\") pod \"kuadrant-operator-controller-manager-86878c769f-7scdw\" (UID: \"da288574-77eb-4bd1-bf0c-9a083b54b2ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.711517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.711445 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:55.862612 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.862588 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw"] Jul 02 23:42:55.864320 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:42:55.864277 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda288574_77eb_4bd1_bf0c_9a083b54b2ba.slice/crio-b4cb3bdb67a3dc497c9ac8a8a354046130db2080748a10731edd3f0ea3df0975 WatchSource:0}: Error finding container b4cb3bdb67a3dc497c9ac8a8a354046130db2080748a10731edd3f0ea3df0975: Status 404 returned error can't find the container with id b4cb3bdb67a3dc497c9ac8a8a354046130db2080748a10731edd3f0ea3df0975 Jul 02 23:42:55.918971 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:55.918946 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" event={"ID":"da288574-77eb-4bd1-bf0c-9a083b54b2ba","Type":"ContainerStarted","Data":"b4cb3bdb67a3dc497c9ac8a8a354046130db2080748a10731edd3f0ea3df0975"} Jul 02 23:42:56.926366 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:56.926328 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" event={"ID":"da288574-77eb-4bd1-bf0c-9a083b54b2ba","Type":"ContainerStarted","Data":"d6d50a83e63601ac8fa73d81527d27204dec3c947f025080c570cea28fc1d950"} Jul 02 23:42:56.926798 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:56.926410 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:42:56.946641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:42:56.946590 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" podStartSLOduration=1.946574214 podStartE2EDuration="1.946574214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:42:56.944477998 +0000 UTC m=+1231.026801923" watchObservedRunningTime="2026-07-02 23:42:56.946574214 +0000 UTC m=+1231.028898138" Jul 02 23:43:07.934979 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:43:07.934950 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7scdw" Jul 02 23:43:54.742957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:43:54.742911 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:45:00.143228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.143145 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-hjxs2"] Jul 02 23:45:00.146845 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.146829 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:00.162523 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.162495 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-hjxs2"] Jul 02 23:45:00.248456 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.248420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqn2\" (UniqueName: \"kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2\") pod \"maas-api-key-cleanup-29717265-hjxs2\" (UID: \"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c\") " pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:00.349762 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.349728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqn2\" (UniqueName: \"kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2\") pod \"maas-api-key-cleanup-29717265-hjxs2\" (UID: \"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c\") " pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:00.358226 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.358199 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqn2\" (UniqueName: \"kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2\") pod \"maas-api-key-cleanup-29717265-hjxs2\" (UID: \"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c\") " pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:00.457608 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.457530 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:00.604440 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.604412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-hjxs2"] Jul 02 23:45:00.606201 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:45:00.606172 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae9e18e_81c4_4ca4_9bfa_70ac44043c5c.slice/crio-476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f WatchSource:0}: Error finding container 476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f: Status 404 returned error can't find the container with id 476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f Jul 02 23:45:00.608013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:00.607991 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:45:01.609811 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:01.609773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" event={"ID":"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c","Type":"ContainerStarted","Data":"476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f"} Jul 02 23:45:03.624813 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:03.624782 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" event={"ID":"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c","Type":"ContainerStarted","Data":"d627f9e2a8fa907c208f250679693242379f2678927980c74484b962cfe7944d"} Jul 02 23:45:03.640472 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:03.640424 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" podStartSLOduration=0.84733434 podStartE2EDuration="3.640408371s" totalImagesPullingTime="2.793074031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:45:03.638568326 +0000 UTC m=+1357.720892251" watchObservedRunningTime="2026-07-02 23:45:03.640408371 +0000 UTC m=+1357.722732294" Jul 02 23:45:04.631925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:04.631889 2561 generic.go:356] "Generic (PLEG): container finished" podID="9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c" containerID="d627f9e2a8fa907c208f250679693242379f2678927980c74484b962cfe7944d" exitCode=0 Jul 02 23:45:04.632300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:04.631982 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" event={"ID":"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c","Type":"ContainerDied","Data":"d627f9e2a8fa907c208f250679693242379f2678927980c74484b962cfe7944d"} Jul 02 23:45:05.772510 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:05.772481 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:05.902176 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:05.902084 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tqn2\" (UniqueName: \"kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2\") pod \"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c\" (UID: \"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c\") " Jul 02 23:45:05.904320 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:05.904278 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2" (OuterVolumeSpecName: "kube-api-access-2tqn2") pod "9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c" (UID: "9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c"). InnerVolumeSpecName "kube-api-access-2tqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:45:06.003162 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:06.003126 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqn2\" (UniqueName: \"kubernetes.io/projected/9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c-kube-api-access-2tqn2\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:45:06.645756 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:06.645728 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" Jul 02 23:45:06.645925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:06.645734 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-hjxs2" event={"ID":"9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c","Type":"ContainerDied","Data":"476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f"} Jul 02 23:45:06.645925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:45:06.645860 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476b33fa261064543964f42d4d3b3dd0db214bbce52cf9ea705b7edc406d2c1f" Jul 02 23:47:26.455052 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:47:26.455027 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:47:26.461840 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:47:26.461814 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:47:39.734168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:47:39.734105 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:46.646556 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:48:46.646521 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:56.423131 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:48:56.423092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:04.679216 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:04.679180 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:09.759175 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:09.759137 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:12.726761 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:12.726729 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:12.825723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:12.825686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:13.234595 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:13.234565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:18.707245 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:18.707152 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:18.807699 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:18.807665 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:18.911900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:18.911874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:19.012941 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:19.012853 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.099423 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.099380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.199710 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.199682 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.434464 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.434383 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:20.434809 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.434795 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c" containerName="cleanup" Jul 02 23:49:20.434852 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.434810 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c" containerName="cleanup" Jul 02 23:49:20.434898 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.434888 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c" containerName="cleanup" Jul 02 23:49:20.438123 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.438106 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.440719 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.440697 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-90a204bb-serving-cert\"" Jul 02 23:49:20.451978 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.451957 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:20.464348 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.464321 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.478954 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.478928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.479089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.478962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.479089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.479063 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rspvh\" (UniqueName: \"kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.563971 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.563926 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.579900 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.579866 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rspvh\" (UniqueName: \"kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.580045 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.579928 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.580045 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.579949 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.580616 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.580596 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.582259 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.582239 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.589959 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.589903 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rspvh\" (UniqueName: \"kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh\") pod \"maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.750066 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.749969 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:20.855124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.855080 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:20.894220 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.894167 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:20.895933 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:49:20.895870 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b92e233_1ebe_43a6_a4e7_da779ee0c1e7.slice/crio-3c6073c8a43bd0eabf3580ab8f4c002d48f2ed44b5976c0f5503e45f0953410d WatchSource:0}: Error finding container 3c6073c8a43bd0eabf3580ab8f4c002d48f2ed44b5976c0f5503e45f0953410d: Status 404 returned error can't find the container with id 3c6073c8a43bd0eabf3580ab8f4c002d48f2ed44b5976c0f5503e45f0953410d Jul 02 23:49:20.955844 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:20.955785 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:21.090365 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:21.090320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" event={"ID":"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7","Type":"ContainerStarted","Data":"3c6073c8a43bd0eabf3580ab8f4c002d48f2ed44b5976c0f5503e45f0953410d"} Jul 02 23:49:22.097663 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:22.097620 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" event={"ID":"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7","Type":"ContainerStarted","Data":"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4"} Jul 02 23:49:22.098130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:22.097754 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:22.115328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:22.115267 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" podStartSLOduration=1.586703565 podStartE2EDuration="2.115248645s" totalImagesPullingTime="528.54508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:49:22.113125467 +0000 UTC m=+1616.195449392" watchObservedRunningTime="2026-07-02 23:49:22.115248645 +0000 UTC m=+1616.197572624" Jul 02 23:49:22.670494 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:22.670460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.237300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.237251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.450998 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.450956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.555555 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.555529 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.575361 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.575326 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:25.575607 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.575577 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" podUID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" containerName="maas-api" containerID="cri-o://496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4" gracePeriod=30 Jul 02 23:49:25.611901 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.611863 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.827762 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.827738 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:25.933575 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.933544 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls\") pod \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " Jul 02 23:49:25.933750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.933609 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca\") pod \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " Jul 02 23:49:25.933750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.933687 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rspvh\" (UniqueName: \"kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh\") pod \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\" (UID: \"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7\") " Jul 02 23:49:25.934010 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.933980 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" (UID: "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:49:25.935789 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.935763 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" (UID: "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:49:25.935904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:25.935826 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh" (OuterVolumeSpecName: "kube-api-access-rspvh") pod "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" (UID: "6b92e233-1ebe-43a6-a4e7-da779ee0c1e7"). InnerVolumeSpecName "kube-api-access-rspvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:49:26.034564 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.034533 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:26.034564 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.034559 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rspvh\" (UniqueName: \"kubernetes.io/projected/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-kube-api-access-rspvh\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:26.034564 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.034571 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:26.121904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.121806 2561 generic.go:356] "Generic (PLEG): container finished" podID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" containerID="496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4" exitCode=0 Jul 02 23:49:26.121904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.121847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" event={"ID":"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7","Type":"ContainerDied","Data":"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4"} Jul 02 23:49:26.121904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.121859 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" Jul 02 23:49:26.121904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.121883 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx" event={"ID":"6b92e233-1ebe-43a6-a4e7-da779ee0c1e7","Type":"ContainerDied","Data":"3c6073c8a43bd0eabf3580ab8f4c002d48f2ed44b5976c0f5503e45f0953410d"} Jul 02 23:49:26.121904 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.121905 2561 scope.go:117] "RemoveContainer" containerID="496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4" Jul 02 23:49:26.131987 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.131969 2561 scope.go:117] "RemoveContainer" containerID="496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4" Jul 02 23:49:26.132230 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:26.132209 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4\": container with ID starting with 496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4 not found: ID does not exist" containerID="496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4" Jul 02 23:49:26.132310 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.132238 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4"} err="failed to get container status \"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4\": rpc error: code = NotFound desc = could not find container \"496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4\": container with ID starting with 496c557092e4d093331cd60725ae729e9fca03608e4b8c9d5d56d058a7ec6ce4 not found: ID does not exist" Jul 02 23:49:26.142677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.142655 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:26.145059 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.145042 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-90a204bb-8595d9b989-ph9tx"] Jul 02 23:49:26.445514 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:26.445440 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" path="/var/lib/kubelet/pods/6b92e233-1ebe-43a6-a4e7-da779ee0c1e7/volumes" Jul 02 23:49:27.004630 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:27.004591 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.312444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.312407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.367598 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.367567 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.625308 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.625199 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.713951 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.713917 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.798415 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.798387 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:31.861341 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:31.861274 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.100818 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.100784 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.159131 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.159102 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:33.159624 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.159606 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" containerName="maas-api" Jul 02 23:49:33.159715 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.159627 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" containerName="maas-api" Jul 02 23:49:33.159778 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.159733 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b92e233-1ebe-43a6-a4e7-da779ee0c1e7" containerName="maas-api" Jul 02 23:49:33.164168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.164144 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.166586 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.166562 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-migrate-914dee6c-serving-cert\"" Jul 02 23:49:33.174883 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.174849 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:33.194384 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.194345 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.297229 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.297186 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.297229 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.297232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.297529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.297367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrq26\" (UniqueName: \"kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.398911 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.398824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.398911 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.398878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.399097 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.398937 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrq26\" (UniqueName: \"kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.399097 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:33.398981 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-migrate-914dee6c-serving-cert: secret "maas-api-e2e-migrate-914dee6c-serving-cert" not found Jul 02 23:49:33.399097 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:33.399052 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls podName:1c2204a9-904b-4c4a-8c58-4a2b42f868d0 nodeName:}" failed. No retries permitted until 2026-07-02 23:49:33.899036651 +0000 UTC m=+1627.981360553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls") pod "maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" (UID: "1c2204a9-904b-4c4a-8c58-4a2b42f868d0") : secret "maas-api-e2e-migrate-914dee6c-serving-cert" not found Jul 02 23:49:33.399652 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.399633 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.407918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.407896 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrq26\" (UniqueName: \"kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.466239 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.466207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.760588 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.760507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.861558 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.861522 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:33.904145 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.904104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:33.906791 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:33.906765 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") pod \"maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:34.077250 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:34.077216 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:34.230726 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:34.230697 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:34.232603 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:49:34.232574 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2204a9_904b_4c4a_8c58_4a2b42f868d0.slice/crio-5eeace12809a8ebde2fa3bbdfe7d856d503682478b56f5ecd91cf54938309570 WatchSource:0}: Error finding container 5eeace12809a8ebde2fa3bbdfe7d856d503682478b56f5ecd91cf54938309570: Status 404 returned error can't find the container with id 5eeace12809a8ebde2fa3bbdfe7d856d503682478b56f5ecd91cf54938309570 Jul 02 23:49:35.078559 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:35.078525 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:35.175849 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:35.175807 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" event={"ID":"1c2204a9-904b-4c4a-8c58-4a2b42f868d0","Type":"ContainerStarted","Data":"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9"} Jul 02 23:49:35.176029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:35.175875 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" event={"ID":"1c2204a9-904b-4c4a-8c58-4a2b42f868d0","Type":"ContainerStarted","Data":"5eeace12809a8ebde2fa3bbdfe7d856d503682478b56f5ecd91cf54938309570"} Jul 02 23:49:35.176029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:35.176004 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:35.190894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:35.190841 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" podStartSLOduration=1.830711596 podStartE2EDuration="2.190824191s" totalImagesPullingTime="360.112595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:49:35.189756058 +0000 UTC m=+1629.272080062" watchObservedRunningTime="2026-07-02 23:49:35.190824191 +0000 UTC m=+1629.273148118" Jul 02 23:49:37.675061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.675027 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:37.842141 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.842093 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:37.916063 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.916026 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:37.916297 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.916260 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" podUID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" containerName="maas-api" containerID="cri-o://529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9" gracePeriod=30 Jul 02 23:49:37.937332 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.937309 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:37.990068 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:37.990036 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:38.174837 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.174816 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:38.196580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.196507 2561 generic.go:356] "Generic (PLEG): container finished" podID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" containerID="529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9" exitCode=0 Jul 02 23:49:38.196580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.196554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" event={"ID":"1c2204a9-904b-4c4a-8c58-4a2b42f868d0","Type":"ContainerDied","Data":"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9"} Jul 02 23:49:38.196580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.196558 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" Jul 02 23:49:38.196814 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.196593 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t" event={"ID":"1c2204a9-904b-4c4a-8c58-4a2b42f868d0","Type":"ContainerDied","Data":"5eeace12809a8ebde2fa3bbdfe7d856d503682478b56f5ecd91cf54938309570"} Jul 02 23:49:38.196814 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.196617 2561 scope.go:117] "RemoveContainer" containerID="529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9" Jul 02 23:49:38.205991 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.205973 2561 scope.go:117] "RemoveContainer" containerID="529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9" Jul 02 23:49:38.206276 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:38.206258 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9\": container with ID starting with 529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9 not found: ID does not exist" containerID="529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9" Jul 02 23:49:38.206362 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.206299 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9"} err="failed to get container status \"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9\": rpc error: code = NotFound desc = could not find container \"529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9\": container with ID starting with 529edce5fe95e535995ca33db553daf9d3ebf39784e5fc28b092f5e79998c4c9 not found: ID does not exist" Jul 02 23:49:38.244764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.244732 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca\") pod \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " Jul 02 23:49:38.244764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.244766 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrq26\" (UniqueName: \"kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26\") pod \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " Jul 02 23:49:38.245002 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.244866 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") pod \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\" (UID: \"1c2204a9-904b-4c4a-8c58-4a2b42f868d0\") " Jul 02 23:49:38.245159 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.245131 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c2204a9-904b-4c4a-8c58-4a2b42f868d0" (UID: "1c2204a9-904b-4c4a-8c58-4a2b42f868d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:49:38.247009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.246989 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c2204a9-904b-4c4a-8c58-4a2b42f868d0" (UID: "1c2204a9-904b-4c4a-8c58-4a2b42f868d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:49:38.247125 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.247106 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26" (OuterVolumeSpecName: "kube-api-access-vrq26") pod "1c2204a9-904b-4c4a-8c58-4a2b42f868d0" (UID: "1c2204a9-904b-4c4a-8c58-4a2b42f868d0"). InnerVolumeSpecName "kube-api-access-vrq26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:49:38.345966 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.345935 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:38.345966 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.345962 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:38.345966 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.345972 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrq26\" (UniqueName: \"kubernetes.io/projected/1c2204a9-904b-4c4a-8c58-4a2b42f868d0-kube-api-access-vrq26\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:38.512469 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.512440 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:38.516046 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:38.516020 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-migrate-914dee6c-6d566897bf-w4n4t"] Jul 02 23:49:40.445609 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:40.445578 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" path="/var/lib/kubelet/pods/1c2204a9-904b-4c4a-8c58-4a2b42f868d0/volumes" Jul 02 23:49:41.009249 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:41.009210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:43.532449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:43.532411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:43.636564 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:43.636494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:43.734644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:43.734618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:43.835457 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:43.835383 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:44.213755 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:44.213663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:44.314031 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:44.313996 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:44.934908 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:44.934879 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:45.033800 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.033773 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:45.128648 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.128615 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:45.129185 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.129161 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" containerName="maas-api" Jul 02 23:49:45.129185 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.129189 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" containerName="maas-api" Jul 02 23:49:45.129413 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.129300 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2204a9-904b-4c4a-8c58-4a2b42f868d0" containerName="maas-api" Jul 02 23:49:45.133863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.133838 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.136205 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.136181 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-caf3ad89-serving-cert\"" Jul 02 23:49:45.147349 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.146471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:45.165271 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.165243 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:45.205511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.205416 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.205511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.205455 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5bs5\" (UniqueName: \"kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.205511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.205484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.306429 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.306396 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.306429 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.306431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5bs5\" (UniqueName: \"kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.306690 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.306477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.306690 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:45.306627 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-caf3ad89-serving-cert: secret "maas-api-e2e-ait-caf3ad89-serving-cert" not found Jul 02 23:49:45.306811 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:45.306704 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls podName:ffa754a5-d42f-46c3-a545-fe1c62236bcc nodeName:}" failed. No retries permitted until 2026-07-02 23:49:45.806689413 +0000 UTC m=+1639.889013315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls") pod "maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" (UID: "ffa754a5-d42f-46c3-a545-fe1c62236bcc") : secret "maas-api-e2e-ait-caf3ad89-serving-cert" not found Jul 02 23:49:45.307146 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.307121 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.315111 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.315086 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5bs5\" (UniqueName: \"kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.452641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.452596 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:45.759128 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.759095 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:45.811678 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.811648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.814302 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.814247 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") pod \"maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:45.861254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:45.861223 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:46.051723 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:46.051690 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:46.190967 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:46.190941 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:46.193079 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:49:46.193050 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa754a5_d42f_46c3_a545_fe1c62236bcc.slice/crio-a08bf0bf11bb5798e9de9a9685d4a041326322ef5c00445d9eddbbe9144461b8 WatchSource:0}: Error finding container a08bf0bf11bb5798e9de9a9685d4a041326322ef5c00445d9eddbbe9144461b8: Status 404 returned error can't find the container with id a08bf0bf11bb5798e9de9a9685d4a041326322ef5c00445d9eddbbe9144461b8 Jul 02 23:49:46.243873 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:46.243838 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" event={"ID":"ffa754a5-d42f-46c3-a545-fe1c62236bcc","Type":"ContainerStarted","Data":"a08bf0bf11bb5798e9de9a9685d4a041326322ef5c00445d9eddbbe9144461b8"} Jul 02 23:49:47.251402 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:47.251366 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" event={"ID":"ffa754a5-d42f-46c3-a545-fe1c62236bcc","Type":"ContainerStarted","Data":"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe"} Jul 02 23:49:47.251885 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:47.251499 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:47.268508 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:47.268459 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" podStartSLOduration=1.862184353 podStartE2EDuration="2.268439894s" totalImagesPullingTime="406.255541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:49:47.265947832 +0000 UTC m=+1641.348271757" watchObservedRunningTime="2026-07-02 23:49:47.268439894 +0000 UTC m=+1641.350763854" Jul 02 23:49:47.347802 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:47.347765 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:50.047628 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.047597 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:50.305661 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.305554 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:50.305834 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.305778 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" podUID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" containerName="maas-api" containerID="cri-o://4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe" gracePeriod=30 Jul 02 23:49:50.328353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.328329 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:50.561756 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.561696 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:50.657098 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.657061 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5bs5\" (UniqueName: \"kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5\") pod \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " Jul 02 23:49:50.657324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.657141 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") pod \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " Jul 02 23:49:50.657324 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.657214 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca\") pod \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\" (UID: \"ffa754a5-d42f-46c3-a545-fe1c62236bcc\") " Jul 02 23:49:50.657648 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.657620 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffa754a5-d42f-46c3-a545-fe1c62236bcc" (UID: "ffa754a5-d42f-46c3-a545-fe1c62236bcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:49:50.659362 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.659336 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffa754a5-d42f-46c3-a545-fe1c62236bcc" (UID: "ffa754a5-d42f-46c3-a545-fe1c62236bcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:49:50.659466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.659395 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5" (OuterVolumeSpecName: "kube-api-access-h5bs5") pod "ffa754a5-d42f-46c3-a545-fe1c62236bcc" (UID: "ffa754a5-d42f-46c3-a545-fe1c62236bcc"). InnerVolumeSpecName "kube-api-access-h5bs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:49:50.758815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.758783 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa754a5-d42f-46c3-a545-fe1c62236bcc-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:50.758815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.758811 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5bs5\" (UniqueName: \"kubernetes.io/projected/ffa754a5-d42f-46c3-a545-fe1c62236bcc-kube-api-access-h5bs5\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:50.758815 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:50.758821 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa754a5-d42f-46c3-a545-fe1c62236bcc-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:49:51.274962 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.274924 2561 generic.go:356] "Generic (PLEG): container finished" podID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" containerID="4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe" exitCode=0 Jul 02 23:49:51.275435 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.274974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" event={"ID":"ffa754a5-d42f-46c3-a545-fe1c62236bcc","Type":"ContainerDied","Data":"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe"} Jul 02 23:49:51.275435 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.274984 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" Jul 02 23:49:51.275435 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.275044 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw" event={"ID":"ffa754a5-d42f-46c3-a545-fe1c62236bcc","Type":"ContainerDied","Data":"a08bf0bf11bb5798e9de9a9685d4a041326322ef5c00445d9eddbbe9144461b8"} Jul 02 23:49:51.275435 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.275066 2561 scope.go:117] "RemoveContainer" containerID="4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe" Jul 02 23:49:51.285240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.285224 2561 scope.go:117] "RemoveContainer" containerID="4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe" Jul 02 23:49:51.285511 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:49:51.285492 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe\": container with ID starting with 4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe not found: ID does not exist" containerID="4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe" Jul 02 23:49:51.285573 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.285520 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe"} err="failed to get container status \"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe\": rpc error: code = NotFound desc = could not find container \"4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe\": container with ID starting with 4ab92d92713d07a358ac479c06921215c6acf1553340e58cf805a32e486c67fe not found: ID does not exist" Jul 02 23:49:51.301629 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.301595 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:51.304159 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:51.304137 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-caf3ad89-56955878dd-fj8tw"] Jul 02 23:49:52.446023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:52.445995 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" path="/var/lib/kubelet/pods/ffa754a5-d42f-46c3-a545-fe1c62236bcc/volumes" Jul 02 23:49:56.408937 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:56.408900 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:56.517638 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:49:56.517600 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:02.139983 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:02.139948 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:02.200191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:02.200150 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:02.339849 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:02.339813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:03.536657 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.536627 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:03.610878 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.610787 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:03.611665 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.611646 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:03.612031 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.612019 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" containerName="maas-api" Jul 02 23:50:03.612089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.612034 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" containerName="maas-api" Jul 02 23:50:03.612136 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.612126 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa754a5-d42f-46c3-a545-fe1c62236bcc" containerName="maas-api" Jul 02 23:50:03.615048 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.615030 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.617272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.617249 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-8b3bd569-serving-cert\"" Jul 02 23:50:03.626222 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.626198 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:03.670874 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.670835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfs5\" (UniqueName: \"kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.671003 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.670884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.671003 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.670975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.682106 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.682078 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:03.771495 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.771457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfs5\" (UniqueName: \"kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.771692 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.771538 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.771692 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.771632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.771811 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:50:03.771741 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-8b3bd569-serving-cert: secret "maas-api-e2e-derive-8b3bd569-serving-cert" not found Jul 02 23:50:03.771869 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:50:03.771832 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls podName:fa05c9f5-8290-4602-86cd-c6bfb36191df nodeName:}" failed. No retries permitted until 2026-07-02 23:50:04.271809844 +0000 UTC m=+1658.354133756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls") pod "maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" (UID: "fa05c9f5-8290-4602-86cd-c6bfb36191df") : secret "maas-api-e2e-derive-8b3bd569-serving-cert" not found Jul 02 23:50:03.772327 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.772278 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:03.774157 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.774133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:03.779952 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:03.779933 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfs5\" (UniqueName: \"kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:04.180987 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.180955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:04.276649 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.276610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:04.279196 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.279169 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") pod \"maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:04.281570 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.281544 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:04.528458 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.528425 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:04.673440 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.673416 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:04.675972 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:50:04.675940 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa05c9f5_8290_4602_86cd_c6bfb36191df.slice/crio-430680a35991b8becc0183b75502314499940710251bd636ab434e6fdb713237 WatchSource:0}: Error finding container 430680a35991b8becc0183b75502314499940710251bd636ab434e6fdb713237: Status 404 returned error can't find the container with id 430680a35991b8becc0183b75502314499940710251bd636ab434e6fdb713237 Jul 02 23:50:04.677539 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:04.677516 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:50:05.360866 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:05.360765 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" event={"ID":"fa05c9f5-8290-4602-86cd-c6bfb36191df","Type":"ContainerStarted","Data":"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c"} Jul 02 23:50:05.360866 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:05.360853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" event={"ID":"fa05c9f5-8290-4602-86cd-c6bfb36191df","Type":"ContainerStarted","Data":"430680a35991b8becc0183b75502314499940710251bd636ab434e6fdb713237"} Jul 02 23:50:05.361101 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:05.360878 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:05.377239 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:05.377179 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" podStartSLOduration=1.9577031200000001 podStartE2EDuration="2.377164976s" totalImagesPullingTime="419.461856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:50:05.374675439 +0000 UTC m=+1659.456999363" watchObservedRunningTime="2026-07-02 23:50:05.377164976 +0000 UTC m=+1659.459488905" Jul 02 23:50:07.024771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:07.024711 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.000683 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.000653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.180003 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.179969 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.280899 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.280864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.313522 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.313487 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:08.313791 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.313765 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" podUID="fa05c9f5-8290-4602-86cd-c6bfb36191df" containerName="maas-api" containerID="cri-o://d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c" gracePeriod=30 Jul 02 23:50:08.337090 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.337066 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.563373 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.563346 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:08.620466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.620436 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca\") pod \"fa05c9f5-8290-4602-86cd-c6bfb36191df\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " Jul 02 23:50:08.620592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.620504 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vfs5\" (UniqueName: \"kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5\") pod \"fa05c9f5-8290-4602-86cd-c6bfb36191df\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " Jul 02 23:50:08.620592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.620571 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") pod \"fa05c9f5-8290-4602-86cd-c6bfb36191df\" (UID: \"fa05c9f5-8290-4602-86cd-c6bfb36191df\") " Jul 02 23:50:08.620988 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.620949 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa05c9f5-8290-4602-86cd-c6bfb36191df" (UID: "fa05c9f5-8290-4602-86cd-c6bfb36191df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:50:08.622877 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.622826 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5" (OuterVolumeSpecName: "kube-api-access-6vfs5") pod "fa05c9f5-8290-4602-86cd-c6bfb36191df" (UID: "fa05c9f5-8290-4602-86cd-c6bfb36191df"). InnerVolumeSpecName "kube-api-access-6vfs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:50:08.623097 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.623064 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa05c9f5-8290-4602-86cd-c6bfb36191df" (UID: "fa05c9f5-8290-4602-86cd-c6bfb36191df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:50:08.721734 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.721691 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa05c9f5-8290-4602-86cd-c6bfb36191df-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:50:08.721734 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.721728 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfs5\" (UniqueName: \"kubernetes.io/projected/fa05c9f5-8290-4602-86cd-c6bfb36191df-kube-api-access-6vfs5\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:50:08.721734 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:08.721742 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa05c9f5-8290-4602-86cd-c6bfb36191df-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:50:09.386130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.386100 2561 generic.go:356] "Generic (PLEG): container finished" podID="fa05c9f5-8290-4602-86cd-c6bfb36191df" containerID="d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c" exitCode=0 Jul 02 23:50:09.386671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.386157 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" Jul 02 23:50:09.386671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.386193 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" event={"ID":"fa05c9f5-8290-4602-86cd-c6bfb36191df","Type":"ContainerDied","Data":"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c"} Jul 02 23:50:09.386671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.386257 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk" event={"ID":"fa05c9f5-8290-4602-86cd-c6bfb36191df","Type":"ContainerDied","Data":"430680a35991b8becc0183b75502314499940710251bd636ab434e6fdb713237"} Jul 02 23:50:09.386671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.386317 2561 scope.go:117] "RemoveContainer" containerID="d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c" Jul 02 23:50:09.396450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.396434 2561 scope.go:117] "RemoveContainer" containerID="d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c" Jul 02 23:50:09.396713 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:50:09.396696 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c\": container with ID starting with d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c not found: ID does not exist" containerID="d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c" Jul 02 23:50:09.396784 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.396720 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c"} err="failed to get container status \"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c\": rpc error: code = NotFound desc = could not find container \"d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c\": container with ID starting with d3156b17c6f6d3be335ccaab0449e2a0c9855bded85a04fa901dd19148f0ca0c not found: ID does not exist" Jul 02 23:50:09.408353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.408330 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:09.411745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.411725 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b3bd569-c9975967c-lnpmk"] Jul 02 23:50:09.754709 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:09.754681 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:10.445085 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:10.445053 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa05c9f5-8290-4602-86cd-c6bfb36191df" path="/var/lib/kubelet/pods/fa05c9f5-8290-4602-86cd-c6bfb36191df/volumes" Jul 02 23:50:13.999211 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:13.999182 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.053492 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:14.053461 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.510459 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:14.510401 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.607845 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:14.607813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.710141 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:14.710106 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:16.117648 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:16.117617 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:21.796544 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:21.796514 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:21.857219 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:21.857191 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:21.930599 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:21.930553 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:22.030483 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:22.030456 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:22.131799 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:22.131711 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:22.501183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:22.501104 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:48.440893 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:48.440804 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:54.020313 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:54.020261 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:54.121576 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:54.121541 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:54.221078 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:50:54.221043 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:10.106421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:10.106376 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:15.614745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:15.614688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:15.670563 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:15.670530 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:15.746165 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:15.746122 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:15.847412 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:15.847376 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:26.328655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:26.328622 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:27.141380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:27.141341 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:32.793346 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:32.793279 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:32.897558 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:32.897504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:32.992491 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:32.992453 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:33.092328 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:33.092210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:33.863210 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:33.863172 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:33.965442 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:33.965414 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:34.062140 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:34.062110 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:35.655443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:35.655413 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:41.425780 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:41.425743 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:41.526937 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:41.526903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:42.093598 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:42.093563 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:47.907061 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:47.907022 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:48.006089 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:48.006056 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:48.140529 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:48.140493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:48.442730 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:48.442694 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:48.727678 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:48.727601 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:51.946385 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:51.946356 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.021169 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.021139 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:51:52.021636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.021616 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa05c9f5-8290-4602-86cd-c6bfb36191df" containerName="maas-api" Jul 02 23:51:52.021636 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.021638 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa05c9f5-8290-4602-86cd-c6bfb36191df" containerName="maas-api" Jul 02 23:51:52.021802 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.021767 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa05c9f5-8290-4602-86cd-c6bfb36191df" containerName="maas-api" Jul 02 23:51:52.023764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.023704 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.025056 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.025037 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.027419 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.027392 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-f6ee9ad4-serving-cert\"" Jul 02 23:51:52.034373 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.034350 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:51:52.097992 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.097962 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.165971 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.165940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.166167 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.166044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgcss\" (UniqueName: \"kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.166167 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.166112 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.266770 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.266744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.266933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.266824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.266933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.266893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgcss\" (UniqueName: \"kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.267021 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:51:52.266974 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-f6ee9ad4-serving-cert: secret "maas-api-e2e-mt-f6ee9ad4-serving-cert" not found Jul 02 23:51:52.267057 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:51:52.267044 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls podName:9407e68a-21f9-4836-9813-d72d1155c704 nodeName:}" failed. No retries permitted until 2026-07-02 23:51:52.767028893 +0000 UTC m=+1766.849352796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls") pod "maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" (UID: "9407e68a-21f9-4836-9813-d72d1155c704") : secret "maas-api-e2e-mt-f6ee9ad4-serving-cert" not found Jul 02 23:51:52.267490 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.267471 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.280868 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.280836 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgcss\" (UniqueName: \"kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.455965 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.455929 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.624622 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.624538 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.724412 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.724379 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:52.771922 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.771888 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.774373 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.774351 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") pod \"maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:52.937674 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:52.937570 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:53.088642 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:53.088614 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:51:53.089533 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:51:53.089509 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9407e68a_21f9_4836_9813_d72d1155c704.slice/crio-639ceb1409fe98ca06695bf451b38d2c0eceb89749eb7e08d50da834c02de45a WatchSource:0}: Error finding container 639ceb1409fe98ca06695bf451b38d2c0eceb89749eb7e08d50da834c02de45a: Status 404 returned error can't find the container with id 639ceb1409fe98ca06695bf451b38d2c0eceb89749eb7e08d50da834c02de45a Jul 02 23:51:53.986577 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:53.986541 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" event={"ID":"9407e68a-21f9-4836-9813-d72d1155c704","Type":"ContainerStarted","Data":"c5fc754b03346a55141f4b201da48e8b46ce184782625618bf9d9987f1f2b40b"} Jul 02 23:51:53.986774 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:53.986593 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" event={"ID":"9407e68a-21f9-4836-9813-d72d1155c704","Type":"ContainerStarted","Data":"639ceb1409fe98ca06695bf451b38d2c0eceb89749eb7e08d50da834c02de45a"} Jul 02 23:51:53.986774 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:53.986700 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:51:54.003563 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:54.003511 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" podStartSLOduration=1.620237864 podStartE2EDuration="2.003497428s" totalImagesPullingTime="383.259564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:51:54.001387573 +0000 UTC m=+1768.083711499" watchObservedRunningTime="2026-07-02 23:51:54.003497428 +0000 UTC m=+1768.085821351" Jul 02 23:51:56.430482 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:56.430440 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:57.087191 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:57.087143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:57.183594 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:57.183560 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:57.508991 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:57.508887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:51:59.999020 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:51:59.998986 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:52:03.209974 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.209941 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:03.309615 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.309578 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:03.605578 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.605543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:03.855023 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.854991 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:52:03.855254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.855225 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" podUID="9407e68a-21f9-4836-9813-d72d1155c704" containerName="maas-api" containerID="cri-o://c5fc754b03346a55141f4b201da48e8b46ce184782625618bf9d9987f1f2b40b" gracePeriod=30 Jul 02 23:52:03.876211 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.876187 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:03.988322 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:03.988207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:04.051349 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.051313 2561 generic.go:356] "Generic (PLEG): container finished" podID="9407e68a-21f9-4836-9813-d72d1155c704" containerID="c5fc754b03346a55141f4b201da48e8b46ce184782625618bf9d9987f1f2b40b" exitCode=0 Jul 02 23:52:04.051526 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.051320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" event={"ID":"9407e68a-21f9-4836-9813-d72d1155c704","Type":"ContainerDied","Data":"c5fc754b03346a55141f4b201da48e8b46ce184782625618bf9d9987f1f2b40b"} Jul 02 23:52:04.108143 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.108083 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:52:04.181200 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.181170 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgcss\" (UniqueName: \"kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss\") pod \"9407e68a-21f9-4836-9813-d72d1155c704\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " Jul 02 23:52:04.181431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.181232 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") pod \"9407e68a-21f9-4836-9813-d72d1155c704\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " Jul 02 23:52:04.181431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.181321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca\") pod \"9407e68a-21f9-4836-9813-d72d1155c704\" (UID: \"9407e68a-21f9-4836-9813-d72d1155c704\") " Jul 02 23:52:04.181713 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.181682 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9407e68a-21f9-4836-9813-d72d1155c704" (UID: "9407e68a-21f9-4836-9813-d72d1155c704"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:52:04.183550 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.183512 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9407e68a-21f9-4836-9813-d72d1155c704" (UID: "9407e68a-21f9-4836-9813-d72d1155c704"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:52:04.183650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.183583 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss" (OuterVolumeSpecName: "kube-api-access-sgcss") pod "9407e68a-21f9-4836-9813-d72d1155c704" (UID: "9407e68a-21f9-4836-9813-d72d1155c704"). InnerVolumeSpecName "kube-api-access-sgcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:52:04.282136 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.282100 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgcss\" (UniqueName: \"kubernetes.io/projected/9407e68a-21f9-4836-9813-d72d1155c704-kube-api-access-sgcss\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:52:04.282136 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.282130 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9407e68a-21f9-4836-9813-d72d1155c704-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:52:04.282136 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:04.282139 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9407e68a-21f9-4836-9813-d72d1155c704-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:52:05.058205 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:05.058165 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" event={"ID":"9407e68a-21f9-4836-9813-d72d1155c704","Type":"ContainerDied","Data":"639ceb1409fe98ca06695bf451b38d2c0eceb89749eb7e08d50da834c02de45a"} Jul 02 23:52:05.058408 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:05.058220 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l" Jul 02 23:52:05.058408 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:05.058242 2561 scope.go:117] "RemoveContainer" containerID="c5fc754b03346a55141f4b201da48e8b46ce184782625618bf9d9987f1f2b40b" Jul 02 23:52:05.074125 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:05.074102 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:52:05.077999 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:05.077979 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-f6ee9ad4-f77cb46d4-qvr9l"] Jul 02 23:52:06.446634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:06.446598 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9407e68a-21f9-4836-9813-d72d1155c704" path="/var/lib/kubelet/pods/9407e68a-21f9-4836-9813-d72d1155c704/volumes" Jul 02 23:52:09.816409 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:09.816371 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:09.869810 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:09.869780 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:10.073002 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:10.072916 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:10.302888 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:10.302856 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:10.426565 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:10.426488 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:10.543329 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:10.543303 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:10.673801 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:10.673768 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:16.522085 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:16.522054 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:16.582161 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:16.582132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:16.660122 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:16.660088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:16.762550 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:16.762514 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:16.864863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:16.864784 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:22.520093 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:22.520009 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:22.583201 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:22.583166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:26.494634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:26.494607 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:52:26.504787 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:26.504765 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:52:28.943882 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:28.943852 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:29.233957 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:29.233883 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:29.669989 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:29.669952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:32.139046 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.139010 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:32.251389 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.251355 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:32.392520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.392431 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:52:32.392972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.392955 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9407e68a-21f9-4836-9813-d72d1155c704" containerName="maas-api" Jul 02 23:52:32.392972 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.392973 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9407e68a-21f9-4836-9813-d72d1155c704" containerName="maas-api" Jul 02 23:52:32.393128 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.393097 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9407e68a-21f9-4836-9813-d72d1155c704" containerName="maas-api" Jul 02 23:52:32.396444 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.396426 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.399948 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.399926 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-cadc708e-serving-cert\"" Jul 02 23:52:32.412896 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.412870 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:52:32.425676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.425643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrcl\" (UniqueName: \"kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.425899 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.425870 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.426007 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.425919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.433879 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.433858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:32.527237 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.527201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrcl\" (UniqueName: \"kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.527237 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.527243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.527474 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.527268 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.527474 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:52:32.527374 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-cadc708e-serving-cert: secret "maas-api-e2e-mt-cadc708e-serving-cert" not found Jul 02 23:52:32.527474 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:52:32.527447 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls podName:d71a6ac1-6294-481f-9b05-8bc06628ad9d nodeName:}" failed. No retries permitted until 2026-07-02 23:52:33.027430812 +0000 UTC m=+1807.109754714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls") pod "maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" (UID: "d71a6ac1-6294-481f-9b05-8bc06628ad9d") : secret "maas-api-e2e-mt-cadc708e-serving-cert" not found Jul 02 23:52:32.527885 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.527868 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.540508 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.536773 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrcl\" (UniqueName: \"kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:32.916899 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:32.916871 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:33.017924 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:33.017891 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:33.031190 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:33.031164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:33.033770 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:33.033741 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") pod \"maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:33.306955 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:33.306914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:33.450712 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:33.450689 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:52:33.452832 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:52:33.452795 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71a6ac1_6294_481f_9b05_8bc06628ad9d.slice/crio-c9c045e3869b37db0aa41615891e1944588426aeb4536e3363a4bb76617075a0 WatchSource:0}: Error finding container c9c045e3869b37db0aa41615891e1944588426aeb4536e3363a4bb76617075a0: Status 404 returned error can't find the container with id c9c045e3869b37db0aa41615891e1944588426aeb4536e3363a4bb76617075a0 Jul 02 23:52:34.242768 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:34.242678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" event={"ID":"d71a6ac1-6294-481f-9b05-8bc06628ad9d","Type":"ContainerStarted","Data":"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491"} Jul 02 23:52:34.242768 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:34.242727 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" event={"ID":"d71a6ac1-6294-481f-9b05-8bc06628ad9d","Type":"ContainerStarted","Data":"c9c045e3869b37db0aa41615891e1944588426aeb4536e3363a4bb76617075a0"} Jul 02 23:52:34.242993 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:34.242795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:34.259125 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:34.259083 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" podStartSLOduration=1.905314728 podStartE2EDuration="2.259070026s" totalImagesPullingTime="353.755298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:52:34.25726442 +0000 UTC m=+1808.339588345" watchObservedRunningTime="2026-07-02 23:52:34.259070026 +0000 UTC m=+1808.341393951" Jul 02 23:52:36.641160 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:36.641132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:36.796703 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:36.796667 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:36.897019 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:36.896926 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:40.254354 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:40.254320 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:52:53.581411 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:53.581373 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:52:53.679355 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:52:53.679324 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:53:48.508496 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:53:48.508406 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:53:48.608990 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:53:48.608956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:41.507348 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:41.507221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:41.721632 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:41.721596 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:55:41.721927 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:41.721867 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" podUID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" containerName="maas-api" containerID="cri-o://d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491" gracePeriod=30 Jul 02 23:55:41.976592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:41.976568 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:55:42.027241 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.027209 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvrcl\" (UniqueName: \"kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl\") pod \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " Jul 02 23:55:42.027453 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.027315 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca\") pod \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " Jul 02 23:55:42.027453 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.027374 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") pod \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\" (UID: \"d71a6ac1-6294-481f-9b05-8bc06628ad9d\") " Jul 02 23:55:42.027670 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.027648 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d71a6ac1-6294-481f-9b05-8bc06628ad9d" (UID: "d71a6ac1-6294-481f-9b05-8bc06628ad9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:55:42.029515 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.029484 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d71a6ac1-6294-481f-9b05-8bc06628ad9d" (UID: "d71a6ac1-6294-481f-9b05-8bc06628ad9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:55:42.029624 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.029537 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl" (OuterVolumeSpecName: "kube-api-access-dvrcl") pod "d71a6ac1-6294-481f-9b05-8bc06628ad9d" (UID: "d71a6ac1-6294-481f-9b05-8bc06628ad9d"). InnerVolumeSpecName "kube-api-access-dvrcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:55:42.128120 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.128035 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrcl\" (UniqueName: \"kubernetes.io/projected/d71a6ac1-6294-481f-9b05-8bc06628ad9d-kube-api-access-dvrcl\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:55:42.128120 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.128060 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a6ac1-6294-481f-9b05-8bc06628ad9d-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:55:42.128120 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.128071 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a6ac1-6294-481f-9b05-8bc06628ad9d-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:55:42.286013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.285970 2561 generic.go:356] "Generic (PLEG): container finished" podID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" containerID="d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491" exitCode=0 Jul 02 23:55:42.286013 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.286013 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" Jul 02 23:55:42.286299 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.286054 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" event={"ID":"d71a6ac1-6294-481f-9b05-8bc06628ad9d","Type":"ContainerDied","Data":"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491"} Jul 02 23:55:42.286299 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.286103 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv" event={"ID":"d71a6ac1-6294-481f-9b05-8bc06628ad9d","Type":"ContainerDied","Data":"c9c045e3869b37db0aa41615891e1944588426aeb4536e3363a4bb76617075a0"} Jul 02 23:55:42.286299 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.286127 2561 scope.go:117] "RemoveContainer" containerID="d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491" Jul 02 23:55:42.300449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.300429 2561 scope.go:117] "RemoveContainer" containerID="d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491" Jul 02 23:55:42.300730 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:55:42.300709 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491\": container with ID starting with d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491 not found: ID does not exist" containerID="d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491" Jul 02 23:55:42.300783 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.300742 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491"} err="failed to get container status \"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491\": rpc error: code = NotFound desc = could not find container \"d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491\": container with ID starting with d3c0355867a4dc3d9f743bf9bc30c4395e651dca58c3e74ffa143a7f01bd6491 not found: ID does not exist" Jul 02 23:55:42.310077 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.310037 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:55:42.313337 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.313313 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-cadc708e-84c6df6cb7-5kdzv"] Jul 02 23:55:42.445130 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.445050 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" path="/var/lib/kubelet/pods/d71a6ac1-6294-481f-9b05-8bc06628ad9d/volumes" Jul 02 23:55:42.749561 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:42.749466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:47.425113 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:47.425077 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:47.475644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:47.475614 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:47.762567 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:47.762454 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:47.963872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:47.963832 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:49.054068 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:49.054009 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:49.151728 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:49.151690 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:49.251905 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:49.251874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:49.958404 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:49.958374 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:50.057915 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:50.057881 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:50.160594 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:50.160560 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:51.550997 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:51.550965 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:57.515867 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:57.515830 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:57.613928 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:57.613899 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:55:58.186714 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:55:58.186678 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:03.770689 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:03.770645 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:03.831071 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:03.831036 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:03.919333 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:03.919269 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:04.023230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:04.023148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:14.482633 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:14.482600 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:14.830243 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:14.830210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:35.777097 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:35.777060 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:41.284029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:41.283990 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:41.345949 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:41.345917 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:41.619442 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:41.619350 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:41.918592 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:41.918498 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:42.207812 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:42.207724 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.254671 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.254623 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.322480 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.322448 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:56:45.322918 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.322904 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" containerName="maas-api" Jul 02 23:56:45.322966 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.322922 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" containerName="maas-api" Jul 02 23:56:45.323025 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.323016 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71a6ac1-6294-481f-9b05-8bc06628ad9d" containerName="maas-api" Jul 02 23:56:45.326316 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.326263 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.328668 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.328647 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-2b5047-serving-cert\"" Jul 02 23:56:45.338260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.338240 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:56:45.338533 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.338509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.387395 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.387365 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pzd\" (UniqueName: \"kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.387395 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.387399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.387645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.387420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.407106 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.407082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.488549 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.488512 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pzd\" (UniqueName: \"kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.488809 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.488782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.488958 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.488829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.488958 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:56:45.488927 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-2b5047-serving-cert: secret "maas-api-e2e-inf-a-2b5047-serving-cert" not found Jul 02 23:56:45.489068 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:56:45.488994 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls podName:2c6158ee-ce88-4a1a-b56c-b2d602f7a95a nodeName:}" failed. No retries permitted until 2026-07-02 23:56:45.988974535 +0000 UTC m=+2060.071298456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls") pod "maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" (UID: "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a") : secret "maas-api-e2e-inf-a-2b5047-serving-cert" not found Jul 02 23:56:45.489505 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.489487 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.498336 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.498276 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pzd\" (UniqueName: \"kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.585813 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.585779 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.986106 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.986005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:45.993243 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.993205 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:45.995835 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:45.995805 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") pod \"maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:46.086045 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:46.086004 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:46.239345 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:46.239248 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:46.387635 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:46.387603 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:56:46.399218 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:46.397742 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:56:46.646807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:46.646766 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" event={"ID":"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a","Type":"ContainerStarted","Data":"f3f3e2d63ada75e5f67c2e65ef59f66479e1ef28545f69ee75a8077d12f1f100"} Jul 02 23:56:47.654032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:47.653940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" event={"ID":"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a","Type":"ContainerStarted","Data":"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6"} Jul 02 23:56:47.654032 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:47.654002 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:47.670037 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:47.669989 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" podStartSLOduration=2.260719143 podStartE2EDuration="2.66997219s" totalImagesPullingTime="409.253047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:56:47.668583524 +0000 UTC m=+2061.750907445" watchObservedRunningTime="2026-07-02 23:56:47.66997219 +0000 UTC m=+2061.752296113" Jul 02 23:56:49.781582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:49.781545 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:50.118147 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:50.118060 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:50.439530 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:50.439453 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:50.865601 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:50.865550 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:53.389474 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.389431 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:53.512446 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.512412 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:53.551750 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.551717 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:56:53.556418 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.556393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.559022 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.558991 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-6edf51-serving-cert\"" Jul 02 23:56:53.565277 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.565009 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:56:53.601009 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.600977 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:53.660164 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.660068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.660359 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.660130 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.660359 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.660322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzn26\" (UniqueName: \"kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.665085 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.665062 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:56:53.761530 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.761494 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.761724 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.761542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzn26\" (UniqueName: \"kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.761724 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.761588 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.762170 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.762146 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.764192 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.764172 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.768872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.768855 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzn26\" (UniqueName: \"kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26\") pod \"maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:53.795699 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.795664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:53.869952 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:53.869925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:54.009956 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.009928 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:56:54.011826 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:56:54.011786 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb836dbc_a056_419b_a442_0951e609e6ba.slice/crio-730c47cdda93575fc4ead4a7a898420f75aaf6d243b7c2cdfe01f1f538063a96 WatchSource:0}: Error finding container 730c47cdda93575fc4ead4a7a898420f75aaf6d243b7c2cdfe01f1f538063a96: Status 404 returned error can't find the container with id 730c47cdda93575fc4ead4a7a898420f75aaf6d243b7c2cdfe01f1f538063a96 Jul 02 23:56:54.110569 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.110535 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:54.211807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.211727 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:56:54.695717 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.695683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" event={"ID":"eb836dbc-a056-419b-a442-0951e609e6ba","Type":"ContainerStarted","Data":"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1"} Jul 02 23:56:54.696134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.695743 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" event={"ID":"eb836dbc-a056-419b-a442-0951e609e6ba","Type":"ContainerStarted","Data":"730c47cdda93575fc4ead4a7a898420f75aaf6d243b7c2cdfe01f1f538063a96"} Jul 02 23:56:54.696134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.695812 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:56:54.713532 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:54.713482 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" podStartSLOduration=1.425641453 podStartE2EDuration="1.713468505s" totalImagesPullingTime="287.827052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:56:54.710603671 +0000 UTC m=+2068.792927595" watchObservedRunningTime="2026-07-02 23:56:54.713468505 +0000 UTC m=+2068.795792428" Jul 02 23:56:57.802307 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:56:57.802244 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:00.707038 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:00.707011 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:57:07.511252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:07.511217 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:07.612773 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:07.612740 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:12.700168 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:12.700135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:14.754706 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:14.754659 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:14.856639 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:14.856611 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:26.532737 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:26.532707 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:57:26.542876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:26.542847 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 02 23:57:37.448733 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:37.448701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:37.547770 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:37.547736 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:57:42.499434 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:57:42.499401 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:01.584545 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:01.584507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:02.439439 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:02.439404 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:02.634024 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:02.633983 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:03.018316 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.018266 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:03.122466 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.122433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:03.218712 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.218679 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:03.316677 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.316580 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:58:03.316843 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.316817 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerName="maas-api" containerID="cri-o://0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6" gracePeriod=30 Jul 02 23:58:03.340163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.340132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:03.660482 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.660440 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.65:8443/health\": dial tcp 10.133.0.65:8443: connect: connection refused" Jul 02 23:58:03.777925 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.777901 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:58:03.920231 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.920128 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca\") pod \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " Jul 02 23:58:03.920458 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.920268 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4pzd\" (UniqueName: \"kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd\") pod \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " Jul 02 23:58:03.920458 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.920351 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") pod \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\" (UID: \"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a\") " Jul 02 23:58:03.920574 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.920530 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" (UID: "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:58:03.920655 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.920637 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:03.922508 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.922488 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" (UID: "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:58:03.922645 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:03.922625 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd" (OuterVolumeSpecName: "kube-api-access-z4pzd") pod "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" (UID: "2c6158ee-ce88-4a1a-b56c-b2d602f7a95a"). InnerVolumeSpecName "kube-api-access-z4pzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:58:04.021132 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.021096 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pzd\" (UniqueName: \"kubernetes.io/projected/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-kube-api-access-z4pzd\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:04.021132 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.021129 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:04.108332 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.108265 2561 generic.go:356] "Generic (PLEG): container finished" podID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerID="0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6" exitCode=0 Jul 02 23:58:04.108332 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.108331 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" Jul 02 23:58:04.108584 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.108356 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" event={"ID":"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a","Type":"ContainerDied","Data":"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6"} Jul 02 23:58:04.108584 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.108411 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk" event={"ID":"2c6158ee-ce88-4a1a-b56c-b2d602f7a95a","Type":"ContainerDied","Data":"f3f3e2d63ada75e5f67c2e65ef59f66479e1ef28545f69ee75a8077d12f1f100"} Jul 02 23:58:04.108584 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.108437 2561 scope.go:117] "RemoveContainer" containerID="0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6" Jul 02 23:58:04.118570 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.118550 2561 scope.go:117] "RemoveContainer" containerID="0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6" Jul 02 23:58:04.118845 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:58:04.118826 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6\": container with ID starting with 0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6 not found: ID does not exist" containerID="0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6" Jul 02 23:58:04.118909 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.118854 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6"} err="failed to get container status \"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6\": rpc error: code = NotFound desc = could not find container \"0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6\": container with ID starting with 0e3351672f3c073bf7a68a9670ca2024477d87b77575e5ecd49b65991c71e2a6 not found: ID does not exist" Jul 02 23:58:04.130071 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.130046 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:58:04.133354 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.133333 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2b5047-5996cc699b-fk2sk"] Jul 02 23:58:04.445797 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:04.445754 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" path="/var/lib/kubelet/pods/2c6158ee-ce88-4a1a-b56c-b2d602f7a95a/volumes" Jul 02 23:58:08.100390 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:08.100354 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:25.200597 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:25.200564 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:33.774982 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:33.774944 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:35.815486 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:35.815434 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:40.184072 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:40.184039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:40.457054 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:40.456970 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:40.686518 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:40.686473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:40.846676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:40.846613 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:41.243252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.243147 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:41.341382 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.341350 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:41.441869 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.441829 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:58:41.442101 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.442074 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" podUID="eb836dbc-a056-419b-a442-0951e609e6ba" containerName="maas-api" containerID="cri-o://b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1" gracePeriod=30 Jul 02 23:58:41.465521 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.465488 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:41.703478 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.703453 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:58:41.745144 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.745087 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls\") pod \"eb836dbc-a056-419b-a442-0951e609e6ba\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " Jul 02 23:58:41.745352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.745156 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzn26\" (UniqueName: \"kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26\") pod \"eb836dbc-a056-419b-a442-0951e609e6ba\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " Jul 02 23:58:41.745352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.745276 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca\") pod \"eb836dbc-a056-419b-a442-0951e609e6ba\" (UID: \"eb836dbc-a056-419b-a442-0951e609e6ba\") " Jul 02 23:58:41.745644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.745619 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb836dbc-a056-419b-a442-0951e609e6ba" (UID: "eb836dbc-a056-419b-a442-0951e609e6ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:58:41.747421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.747400 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26" (OuterVolumeSpecName: "kube-api-access-nzn26") pod "eb836dbc-a056-419b-a442-0951e609e6ba" (UID: "eb836dbc-a056-419b-a442-0951e609e6ba"). InnerVolumeSpecName "kube-api-access-nzn26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:58:41.747421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.747404 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb836dbc-a056-419b-a442-0951e609e6ba" (UID: "eb836dbc-a056-419b-a442-0951e609e6ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:58:41.846416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.846325 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb836dbc-a056-419b-a442-0951e609e6ba-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:41.846416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.846362 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb836dbc-a056-419b-a442-0951e609e6ba-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:41.846416 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:41.846377 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzn26\" (UniqueName: \"kubernetes.io/projected/eb836dbc-a056-419b-a442-0951e609e6ba-kube-api-access-nzn26\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:58:42.322499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.322465 2561 generic.go:356] "Generic (PLEG): container finished" podID="eb836dbc-a056-419b-a442-0951e609e6ba" containerID="b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1" exitCode=0 Jul 02 23:58:42.322870 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.322514 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" Jul 02 23:58:42.322870 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.322552 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" event={"ID":"eb836dbc-a056-419b-a442-0951e609e6ba","Type":"ContainerDied","Data":"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1"} Jul 02 23:58:42.322870 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.322634 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5" event={"ID":"eb836dbc-a056-419b-a442-0951e609e6ba","Type":"ContainerDied","Data":"730c47cdda93575fc4ead4a7a898420f75aaf6d243b7c2cdfe01f1f538063a96"} Jul 02 23:58:42.322870 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.322659 2561 scope.go:117] "RemoveContainer" containerID="b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1" Jul 02 23:58:42.333140 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.333123 2561 scope.go:117] "RemoveContainer" containerID="b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1" Jul 02 23:58:42.333464 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:58:42.333436 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1\": container with ID starting with b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1 not found: ID does not exist" containerID="b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1" Jul 02 23:58:42.333525 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.333465 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1"} err="failed to get container status \"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1\": rpc error: code = NotFound desc = could not find container \"b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1\": container with ID starting with b1f3a94d8f0c3ad53731d62a992f8de9bf14d83fdce3c6de75af2c0709b497c1 not found: ID does not exist" Jul 02 23:58:42.345393 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.345369 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:58:42.347703 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.347684 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-6edf51-5b9678fdf8-pnrv5"] Jul 02 23:58:42.445770 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:42.445737 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb836dbc-a056-419b-a442-0951e609e6ba" path="/var/lib/kubelet/pods/eb836dbc-a056-419b-a442-0951e609e6ba/volumes" Jul 02 23:58:46.971037 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:46.970946 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:47.083981 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:47.083951 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:47.241059 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:47.240970 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:47.463449 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:47.463421 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:47.805715 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:47.805685 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:48.142278 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142205 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:58:48.142689 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142672 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb836dbc-a056-419b-a442-0951e609e6ba" containerName="maas-api" Jul 02 23:58:48.142740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142692 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb836dbc-a056-419b-a442-0951e609e6ba" containerName="maas-api" Jul 02 23:58:48.142740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142708 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerName="maas-api" Jul 02 23:58:48.142740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142714 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerName="maas-api" Jul 02 23:58:48.142844 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142786 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6158ee-ce88-4a1a-b56c-b2d602f7a95a" containerName="maas-api" Jul 02 23:58:48.142844 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.142794 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb836dbc-a056-419b-a442-0951e609e6ba" containerName="maas-api" Jul 02 23:58:48.147399 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.147373 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.150300 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.150253 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a89e2b-openshift-default-dockercfg-84jql\"" Jul 02 23:58:48.159744 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.159713 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:58:48.174650 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.174621 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:48.203932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.203897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.203932 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.203933 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.203970 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.203996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nznh\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.204019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.204043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.204111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.204142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.204163 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.204164 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.304934 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nznh\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304949 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.304979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305197 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305087 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305409 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305341 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305409 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305352 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305517 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305489 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.305708 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305685 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.306069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.305947 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.306696 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.306675 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:48.307606 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.307583 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.307853 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.307834 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.312555 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.312533 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.313059 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.313037 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nznh\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh\") pod \"e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.461847 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.461766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:48.606999 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:48.606963 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:58:48.609163 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:58:48.609133 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3362379b_8420_4b23_8ad5_0b63d88e635c.slice/crio-7ae60194c8924bf478a5291fc4219853959061b3dcb494354e4d25cc7d52661a WatchSource:0}: Error finding container 7ae60194c8924bf478a5291fc4219853959061b3dcb494354e4d25cc7d52661a: Status 404 returned error can't find the container with id 7ae60194c8924bf478a5291fc4219853959061b3dcb494354e4d25cc7d52661a Jul 02 23:58:49.370876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:49.370840 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerStarted","Data":"7ae60194c8924bf478a5291fc4219853959061b3dcb494354e4d25cc7d52661a"} Jul 02 23:58:51.380538 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:51.380504 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:58:51.380783 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:51.380576 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:58:51.380783 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:51.380604 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:58:52.390330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:52.390271 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerStarted","Data":"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9"} Jul 02 23:58:52.411252 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:52.411203 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podStartSLOduration=1.6420719030000002 podStartE2EDuration="4.411189571s" totalImagesPullingTime="2.769117668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:58:52.40882052 +0000 UTC m=+2186.491144444" watchObservedRunningTime="2026-07-02 23:58:52.411189571 +0000 UTC m=+2186.493513496" Jul 02 23:58:52.462382 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:52.462352 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:52.467728 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:52.467701 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:53.396069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:53.396027 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:53.396946 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:53.396927 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:58:54.401982 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:54.401949 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:54.491939 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:54.491906 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:54.556851 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:54.556819 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:54.949788 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:54.949757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:55.050857 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:55.050825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:58.905331 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:58.905265 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:59.374801 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:59.374767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:58:59.681732 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:58:59.681652 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:00.111175 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:00.111138 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:02.901307 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:02.901225 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:02.989220 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:02.989175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:03.012407 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.012371 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:03.025511 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.025480 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:03.025842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.025785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.030497 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.030472 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-18475e-serving-cert\"" Jul 02 23:59:03.049067 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.049037 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:03.153408 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.153261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7ss\" (UniqueName: \"kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.153408 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.153397 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.153627 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.153456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.254784 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.254752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.254905 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.254807 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.254905 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.254883 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7ss\" (UniqueName: \"kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.254999 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:59:03.254915 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-18475e-serving-cert: secret "maas-api-e2e-api-b-18475e-serving-cert" not found Jul 02 23:59:03.255053 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:59:03.254996 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls podName:7ca600cb-8fad-478b-abaa-c3d5bcee65d1 nodeName:}" failed. No retries permitted until 2026-07-02 23:59:03.754972022 +0000 UTC m=+2197.837295923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls") pod "maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" (UID: "7ca600cb-8fad-478b-abaa-c3d5bcee65d1") : secret "maas-api-e2e-api-b-18475e-serving-cert" not found Jul 02 23:59:03.255485 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.255468 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.262807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.262787 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7ss\" (UniqueName: \"kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.322949 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.322905 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:03.646630 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.646598 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:03.747998 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.747968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:03.760208 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.760176 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.762959 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.762928 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") pod \"maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:03.939022 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:03.938925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:04.103124 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:04.103090 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:04.105614 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:59:04.105578 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca600cb_8fad_478b_abaa_c3d5bcee65d1.slice/crio-5a4636df7f0e7d0553b438084b405e05027ffd166f69ee79f5557f5904778556 WatchSource:0}: Error finding container 5a4636df7f0e7d0553b438084b405e05027ffd166f69ee79f5557f5904778556: Status 404 returned error can't find the container with id 5a4636df7f0e7d0553b438084b405e05027ffd166f69ee79f5557f5904778556 Jul 02 23:59:04.463731 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:04.463581 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" event={"ID":"7ca600cb-8fad-478b-abaa-c3d5bcee65d1","Type":"ContainerStarted","Data":"5a4636df7f0e7d0553b438084b405e05027ffd166f69ee79f5557f5904778556"} Jul 02 23:59:05.470474 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:05.470418 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" event={"ID":"7ca600cb-8fad-478b-abaa-c3d5bcee65d1","Type":"ContainerStarted","Data":"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf"} Jul 02 23:59:05.470864 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:05.470505 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:05.489755 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:05.489696 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" podStartSLOduration=3.149864127 podStartE2EDuration="3.489676374s" totalImagesPullingTime="339.812247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:59:05.484829721 +0000 UTC m=+2199.567153647" watchObservedRunningTime="2026-07-02 23:59:05.489676374 +0000 UTC m=+2199.572000300" Jul 02 23:59:06.481096 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.481007 2561 generic.go:356] "Generic (PLEG): container finished" podID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerID="3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9" exitCode=0 Jul 02 23:59:06.481096 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.481083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerDied","Data":"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9"} Jul 02 23:59:06.481641 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.481624 2561 scope.go:117] "RemoveContainer" containerID="3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9" Jul 02 23:59:06.482589 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.482559 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:06.482684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.482632 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:06.482684 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:06.482661 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:07.415380 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:07.415344 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:07.491520 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:07.491484 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerStarted","Data":"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72"} Jul 02 23:59:08.396117 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:08.396079 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:08.462523 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:08.462492 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:08.463740 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:08.463715 2561 patch_prober.go:28] interesting pod/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 02 23:59:08.463860 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:08.463762 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 02 23:59:09.462630 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:09.462592 2561 patch_prober.go:28] interesting pod/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 02 23:59:09.463033 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:09.462646 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 02 23:59:10.462102 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:10.462069 2561 patch_prober.go:28] interesting pod/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 02 23:59:10.462274 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:10.462119 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 02 23:59:11.466461 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:11.466432 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:11.467712 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:11.467689 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:11.487484 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:11.487456 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:15.497472 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:15.497435 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:15.765771 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:15.765735 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:15.867646 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:15.867604 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:15.971742 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:15.971713 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:16.068502 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:16.068409 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:21.632461 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:21.632430 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:21.686461 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:21.686431 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:21.870372 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:21.870340 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:59:21.870585 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:21.870562 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" containerID="cri-o://90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72" gracePeriod=30 Jul 02 23:59:21.892689 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:21.892623 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.111531 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.111500 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.278717 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.278688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.661546 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.661467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.762402 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.762350 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.862810 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.862775 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:22.938100 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.938017 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:22.938323 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.938278 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" podUID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" containerName="maas-api" containerID="cri-o://f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf" gracePeriod=30 Jul 02 23:59:22.965610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:22.965586 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:23.196983 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.196913 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:23.253565 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.253535 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml7ss\" (UniqueName: \"kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss\") pod \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " Jul 02 23:59:23.253752 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.253587 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") pod \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " Jul 02 23:59:23.253752 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.253693 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca\") pod \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\" (UID: \"7ca600cb-8fad-478b-abaa-c3d5bcee65d1\") " Jul 02 23:59:23.254056 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.254028 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ca600cb-8fad-478b-abaa-c3d5bcee65d1" (UID: "7ca600cb-8fad-478b-abaa-c3d5bcee65d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:59:23.255801 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.255780 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ca600cb-8fad-478b-abaa-c3d5bcee65d1" (UID: "7ca600cb-8fad-478b-abaa-c3d5bcee65d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:59:23.255801 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.255785 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss" (OuterVolumeSpecName: "kube-api-access-ml7ss") pod "7ca600cb-8fad-478b-abaa-c3d5bcee65d1" (UID: "7ca600cb-8fad-478b-abaa-c3d5bcee65d1"). InnerVolumeSpecName "kube-api-access-ml7ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:59:23.354613 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.354581 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:23.354613 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.354610 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7ss\" (UniqueName: \"kubernetes.io/projected/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-kube-api-access-ml7ss\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:23.354613 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.354620 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca600cb-8fad-478b-abaa-c3d5bcee65d1-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:23.594215 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.594175 2561 generic.go:356] "Generic (PLEG): container finished" podID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" containerID="f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf" exitCode=0 Jul 02 23:59:23.594381 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.594225 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" Jul 02 23:59:23.594381 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.594235 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" event={"ID":"7ca600cb-8fad-478b-abaa-c3d5bcee65d1","Type":"ContainerDied","Data":"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf"} Jul 02 23:59:23.594381 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.594306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z" event={"ID":"7ca600cb-8fad-478b-abaa-c3d5bcee65d1","Type":"ContainerDied","Data":"5a4636df7f0e7d0553b438084b405e05027ffd166f69ee79f5557f5904778556"} Jul 02 23:59:23.594381 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.594337 2561 scope.go:117] "RemoveContainer" containerID="f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf" Jul 02 23:59:23.604105 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.604088 2561 scope.go:117] "RemoveContainer" containerID="f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf" Jul 02 23:59:23.604366 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:59:23.604343 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf\": container with ID starting with f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf not found: ID does not exist" containerID="f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf" Jul 02 23:59:23.604440 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.604372 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf"} err="failed to get container status \"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf\": rpc error: code = NotFound desc = could not find container \"f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf\": container with ID starting with f76d42cb354d89d00ca2a1dd5cd6b945eecf2fdd2857b0573517d3dbb16b8caf not found: ID does not exist" Jul 02 23:59:23.615546 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.615522 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:23.621183 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:23.621163 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-18475e-5fd56458dd-lgb8z"] Jul 02 23:59:24.446260 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:24.446227 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" path="/var/lib/kubelet/pods/7ca600cb-8fad-478b-abaa-c3d5bcee65d1/volumes" Jul 02 23:59:26.469502 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:26.469470 2561 patch_prober.go:28] interesting pod/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 23:59:26.469865 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:26.469533 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 23:59:27.143029 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.142964 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:27.188452 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188417 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188486 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188517 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188634 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188561 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188637 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nznh\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188669 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188697 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188723 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188721 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:59:27.188807 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.188747 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy\") pod \"3362379b-8420-4b23-8ad5-0b63d88e635c\" (UID: \"3362379b-8420-4b23-8ad5-0b63d88e635c\") " Jul 02 23:59:27.189103 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189047 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:59:27.189166 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189149 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-credential-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.189223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189166 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.189409 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189389 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:59:27.189580 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189412 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:59:27.189738 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.189714 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:59:27.191582 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.191542 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:59:27.191691 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.191585 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 23:59:27.192028 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.191999 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 02 23:59:27.192133 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.192101 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh" (OuterVolumeSpecName: "kube-api-access-7nznh") pod "3362379b-8420-4b23-8ad5-0b63d88e635c" (UID: "3362379b-8420-4b23-8ad5-0b63d88e635c"). InnerVolumeSpecName "kube-api-access-7nznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:59:27.290296 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290244 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-token\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290296 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290272 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-data\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290313 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-workload-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290326 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nznh\" (UniqueName: \"kubernetes.io/projected/3362379b-8420-4b23-8ad5-0b63d88e635c-kube-api-access-7nznh\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290339 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-podinfo\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290350 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3362379b-8420-4b23-8ad5-0b63d88e635c-istiod-ca-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.290499 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.290361 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3362379b-8420-4b23-8ad5-0b63d88e635c-istio-envoy\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 02 23:59:27.623453 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.623420 2561 generic.go:356] "Generic (PLEG): container finished" podID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerID="90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72" exitCode=0 Jul 02 23:59:27.623872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.623466 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" Jul 02 23:59:27.623872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.623514 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerDied","Data":"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72"} Jul 02 23:59:27.623872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.623573 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg" event={"ID":"3362379b-8420-4b23-8ad5-0b63d88e635c","Type":"ContainerDied","Data":"7ae60194c8924bf478a5291fc4219853959061b3dcb494354e4d25cc7d52661a"} Jul 02 23:59:27.623872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.623604 2561 scope.go:117] "RemoveContainer" containerID="90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72" Jul 02 23:59:27.633063 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.633044 2561 scope.go:117] "RemoveContainer" containerID="3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9" Jul 02 23:59:27.641745 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.641722 2561 scope.go:117] "RemoveContainer" containerID="90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72" Jul 02 23:59:27.642130 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:59:27.642100 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72\": container with ID starting with 90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72 not found: ID does not exist" containerID="90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72" Jul 02 23:59:27.642230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.642140 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72"} err="failed to get container status \"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72\": rpc error: code = NotFound desc = could not find container \"90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72\": container with ID starting with 90142bfb7d0240c450278a1bd9efa998659ed300490d53523af79db620736a72 not found: ID does not exist" Jul 02 23:59:27.642230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.642163 2561 scope.go:117] "RemoveContainer" containerID="3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9" Jul 02 23:59:27.642520 ip-10-0-130-115 kubenswrapper[2561]: E0702 23:59:27.642494 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9\": container with ID starting with 3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9 not found: ID does not exist" containerID="3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9" Jul 02 23:59:27.642569 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.642532 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9"} err="failed to get container status \"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9\": rpc error: code = NotFound desc = could not find container \"3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9\": container with ID starting with 3062498b69e2b12873b85f594f08296b9da8eadbdbb88057681539e51bb18da9 not found: ID does not exist" Jul 02 23:59:27.645353 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.645331 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:59:27.647795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:27.647773 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a89e2b-openshift-default-7468d54c89-rrzvg"] Jul 02 23:59:28.365753 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:28.365718 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:28.447214 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:28.447172 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" path="/var/lib/kubelet/pods/3362379b-8420-4b23-8ad5-0b63d88e635c/volumes" Jul 02 23:59:28.454223 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:28.454192 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:28.605825 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:28.605792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:28.822203 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:28.822167 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:29.164749 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.164664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:29.436395 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436318 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 02 23:59:29.436715 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436702 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" containerName="maas-api" Jul 02 23:59:29.436759 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436717 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" containerName="maas-api" Jul 02 23:59:29.436759 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436731 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:29.436759 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436737 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:29.436850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436811 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca600cb-8fad-478b-abaa-c3d5bcee65d1" containerName="maas-api" Jul 02 23:59:29.436850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.436822 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:29.439098 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.439078 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.441432 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.441415 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c1893b-openshift-default-dockercfg-xfcz6\"" Jul 02 23:59:29.453576 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.453553 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 02 23:59:29.455708 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.455683 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:29.513625 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513706 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513744 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513782 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513776 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513794 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mhj\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.513926 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.513861 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614644 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614724 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614756 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mhj\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.614824 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.614838 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.615125 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615195 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.615133 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615387 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.615215 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615552 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.615525 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.615850 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.615605 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.617077 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.617046 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.617609 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.617586 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.623151 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.623129 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.623547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.623526 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mhj\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj\") pod \"e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.749602 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.749501 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:29.894519 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.894481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:29.914594 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.914564 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 02 23:59:29.916252 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:59:29.916213 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea23c676_d4d7_4dde_9627_671334217ee4.slice/crio-858e2e389d67f6f49e63f2e37f33a09b30ba7e249cc735b3f6551a55fc0a92fe WatchSource:0}: Error finding container 858e2e389d67f6f49e63f2e37f33a09b30ba7e249cc735b3f6551a55fc0a92fe: Status 404 returned error can't find the container with id 858e2e389d67f6f49e63f2e37f33a09b30ba7e249cc735b3f6551a55fc0a92fe Jul 02 23:59:29.918924 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.918879 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:29.919017 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.918958 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:29.919064 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:29.919025 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:30.645240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.645200 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerStarted","Data":"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469"} Jul 02 23:59:30.645471 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.645255 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerStarted","Data":"858e2e389d67f6f49e63f2e37f33a09b30ba7e249cc735b3f6551a55fc0a92fe"} Jul 02 23:59:30.666676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.666621 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podStartSLOduration=1.666603173 podStartE2EDuration="1.666603173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:59:30.665520517 +0000 UTC m=+2224.747844441" watchObservedRunningTime="2026-07-02 23:59:30.666603173 +0000 UTC m=+2224.748927098" Jul 02 23:59:30.749946 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.749914 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:30.751704 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.751683 2561 patch_prober.go:28] interesting pod/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 02 23:59:30.751770 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:30.751728 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 02 23:59:31.754779 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:31.754741 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:32.642676 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:32.642639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:32.657272 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:32.657242 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:32.658254 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:32.658235 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:32.731427 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:32.731378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:32.796042 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:32.796012 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:33.182173 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:33.182140 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:33.284461 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:33.284412 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:37.134651 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.134616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:37.468826 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.468741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:37.739472 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.739376 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 02 23:59:37.739889 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.739872 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:37.739966 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.739893 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:37.740027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.739969 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3362379b-8420-4b23-8ad5-0b63d88e635c" containerName="istio-proxy" Jul 02 23:59:37.743443 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.743411 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.745876 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.745856 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c04a64-openshift-default-dockercfg-tt2vt\"" Jul 02 23:59:37.755228 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.755201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 02 23:59:37.769547 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.769505 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:37.869820 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.869784 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:37.892920 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.892884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.892940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.892968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnvb\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893066 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893103 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893179 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893158 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.893560 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.893317 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994531 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994436 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994531 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994531 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnvb\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994621 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994674 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994704 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994741 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.994830 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994787 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995108 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994874 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995108 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.994890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995108 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.995065 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995108 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.995079 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995247 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.995143 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.995881 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.995858 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.997069 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.997043 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:37.997326 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:37.997305 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:38.002138 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.002108 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:38.002250 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.002234 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnvb\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb\") pod \"e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:38.055352 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.055316 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:38.197465 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.197433 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 02 23:59:38.199170 ip-10-0-130-115 kubenswrapper[2561]: W0702 23:59:38.199140 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2337058d_78c8_41ea_bac0_e1d768e5fcf5.slice/crio-969aca8e2c0e20b343d8bd874911a96302b26d5fba9cddcb90efd34f992ba59e WatchSource:0}: Error finding container 969aca8e2c0e20b343d8bd874911a96302b26d5fba9cddcb90efd34f992ba59e: Status 404 returned error can't find the container with id 969aca8e2c0e20b343d8bd874911a96302b26d5fba9cddcb90efd34f992ba59e Jul 02 23:59:38.201421 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.201390 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:38.201498 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.201454 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:38.201498 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.201482 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:38.693842 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.693803 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerStarted","Data":"29a461b9cfae7b302306c93203fed8f4608699258a08b090c54e878ae6fb1be4"} Jul 02 23:59:38.694057 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.693859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerStarted","Data":"969aca8e2c0e20b343d8bd874911a96302b26d5fba9cddcb90efd34f992ba59e"} Jul 02 23:59:38.716027 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:38.715977 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podStartSLOduration=1.71596086 podStartE2EDuration="1.71596086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:59:38.713150177 +0000 UTC m=+2232.795474102" watchObservedRunningTime="2026-07-02 23:59:38.71596086 +0000 UTC m=+2232.798284783" Jul 02 23:59:39.056348 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:39.056318 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:39.061305 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:39.061265 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:39.704883 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:39.704855 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:39.708866 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:39.708842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:40.777325 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:40.777265 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:40.862988 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:40.862957 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:40.925841 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:40.925783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:41.282126 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:41.282090 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:41.454801 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:41.454774 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:41.556980 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:41.556885 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:42.349330 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:42.349261 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:45.254149 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:45.254117 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:46.751412 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.751324 2561 generic.go:356] "Generic (PLEG): container finished" podID="ea23c676-d4d7-4dde-9627-671334217ee4" containerID="0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469" exitCode=0 Jul 02 23:59:46.751819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.751401 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerDied","Data":"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469"} Jul 02 23:59:46.751819 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.751816 2561 scope.go:117] "RemoveContainer" containerID="0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469" Jul 02 23:59:46.752836 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.752811 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:46.752894 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.752874 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:46.752933 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:46.752904 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:47.657863 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:47.657778 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:47.759448 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:47.759417 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerStarted","Data":"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7"} Jul 02 23:59:48.750134 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:48.750096 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:48.751397 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:48.751368 2561 patch_prober.go:28] interesting pod/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 02 23:59:48.751532 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:48.751424 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 02 23:59:49.750143 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:49.750100 2561 patch_prober.go:28] interesting pod/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 02 23:59:49.750670 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:49.750178 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 02 23:59:50.750696 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:50.750662 2561 patch_prober.go:28] interesting pod/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 02 23:59:50.751122 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:50.750718 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 02 23:59:51.755107 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:51.755068 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:51.755549 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:51.755531 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:51.756232 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:51.756211 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 02 23:59:53.800166 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.800133 2561 generic.go:356] "Generic (PLEG): container finished" podID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerID="29a461b9cfae7b302306c93203fed8f4608699258a08b090c54e878ae6fb1be4" exitCode=0 Jul 02 23:59:53.800610 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.800207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerDied","Data":"29a461b9cfae7b302306c93203fed8f4608699258a08b090c54e878ae6fb1be4"} Jul 02 23:59:53.800670 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.800654 2561 scope.go:117] "RemoveContainer" containerID="29a461b9cfae7b302306c93203fed8f4608699258a08b090c54e878ae6fb1be4" Jul 02 23:59:53.801719 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.801690 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:53.801795 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.801778 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:53.801847 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:53.801835 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 23:59:54.649114 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:54.649079 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:54.705431 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:54.705386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:54.748717 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:54.748677 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 02 23:59:54.824802 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:54.824762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerStarted","Data":"6e4b14d413bc328c931c19b93fa4922ca49e83a049b1588c4d413f1ecd1898b3"} Jul 02 23:59:55.056541 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:55.056500 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:55.057764 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:55.057732 2561 patch_prober.go:28] interesting pod/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 02 23:59:55.058240 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:55.058210 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 02 23:59:56.056351 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:56.056312 2561 patch_prober.go:28] interesting pod/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 02 23:59:56.056762 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:56.056372 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 02 23:59:57.056253 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:57.056218 2561 patch_prober.go:28] interesting pod/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 02 23:59:57.056450 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:57.056309 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 02 23:59:58.056303 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:58.056260 2561 patch_prober.go:28] interesting pod/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 02 23:59:58.056489 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:58.056358 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 02 23:59:59.059872 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:59.059830 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:59.060230 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:59.060119 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 02 23:59:59.060757 ip-10-0-130-115 kubenswrapper[2561]: I0702 23:59:59.060740 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 03 00:00:03.120552 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:00:03.120510 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:00:03.579252 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:00:03.579211 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:00:09.375510 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:00:09.375481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:00:55.398633 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:00:55.398579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:01:39.368596 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:01:39.368515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:01:39.418361 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:01:39.418321 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:26.570486 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:26.570458 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 03 00:02:26.581091 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:26.581069 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 03 00:02:38.075496 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:38.075460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:38.176483 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:38.176436 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:46.135512 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:46.135473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:46.247650 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:46.247616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:47.197609 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:47.197577 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:02:54.460112 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:02:54.460023 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:03:22.185570 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:03:22.185536 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:24.507387 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:24.507268 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:25.299042 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:25.298999 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:31.199540 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:31.199500 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:54.201793 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:54.201762 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:55.318944 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:55.318907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:04:55.419853 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:04:55.419818 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:03.677395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:03.677347 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:03.778381 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:03.778355 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:39.424091 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:39.424061 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:40.282801 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:40.282763 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:41.396483 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:41.396452 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:41.840443 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:41.840408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:41.948844 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:41.948809 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:42.110764 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:42.110664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:47.242140 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:47.242107 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:05:48.689224 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:05:48.689131 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:14.665023 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:14.664992 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:23.771203 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:23.771169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:24.046466 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:24.046381 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 03 00:06:24.046642 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:24.046596 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" containerID="cri-o://2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7" gracePeriod=30 Jul 03 00:06:24.065630 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:24.065596 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:24.337686 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:24.337604 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:24.649183 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:24.649092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:25.056941 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:25.056907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:25.157156 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:25.157124 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:25.270461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:25.270432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:29.317486 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.317426 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 03 00:06:29.403797 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403766 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403808 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403827 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403864 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403882 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403946 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.403964 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404001 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7mhj\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj\") pod \"ea23c676-d4d7-4dde-9627-671334217ee4\" (UID: \"ea23c676-d4d7-4dde-9627-671334217ee4\") " Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404161 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404193 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404349 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404361 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404380 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-workload-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.404461 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.404380 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 00:06:29.405362 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.405335 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:29.406355 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.406335 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 00:06:29.406500 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.406479 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj" (OuterVolumeSpecName: "kube-api-access-c7mhj") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "kube-api-access-c7mhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 00:06:29.406773 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.406746 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:29.406869 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.406768 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea23c676-d4d7-4dde-9627-671334217ee4" (UID: "ea23c676-d4d7-4dde-9627-671334217ee4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 00:06:29.505116 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505072 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-envoy\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505116 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505113 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-istio-token\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505133 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mhj\" (UniqueName: \"kubernetes.io/projected/ea23c676-d4d7-4dde-9627-671334217ee4-kube-api-access-c7mhj\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505147 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea23c676-d4d7-4dde-9627-671334217ee4-istio-podinfo\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505160 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea23c676-d4d7-4dde-9627-671334217ee4-istiod-ca-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505174 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-credential-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:29.505395 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:29.505194 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea23c676-d4d7-4dde-9627-671334217ee4-istio-data\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:30.115375 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.115343 2561 generic.go:356] "Generic (PLEG): container finished" podID="ea23c676-d4d7-4dde-9627-671334217ee4" containerID="2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7" exitCode=0 Jul 03 00:06:30.115563 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.115407 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" Jul 03 00:06:30.115563 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.115432 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerDied","Data":"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7"} Jul 03 00:06:30.115563 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.115512 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx" event={"ID":"ea23c676-d4d7-4dde-9627-671334217ee4","Type":"ContainerDied","Data":"858e2e389d67f6f49e63f2e37f33a09b30ba7e249cc735b3f6551a55fc0a92fe"} Jul 03 00:06:30.115563 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.115543 2561 scope.go:117] "RemoveContainer" containerID="2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7" Jul 03 00:06:30.126112 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.126095 2561 scope.go:117] "RemoveContainer" containerID="0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469" Jul 03 00:06:30.135019 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.134998 2561 scope.go:117] "RemoveContainer" containerID="2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7" Jul 03 00:06:30.135332 ip-10-0-130-115 kubenswrapper[2561]: E0703 00:06:30.135309 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7\": container with ID starting with 2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7 not found: ID does not exist" containerID="2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7" Jul 03 00:06:30.135388 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.135342 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7"} err="failed to get container status \"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7\": rpc error: code = NotFound desc = could not find container \"2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7\": container with ID starting with 2b43141abdfc10e95740dad77080720375b2d5b1d937c0a68cc94227984193c7 not found: ID does not exist" Jul 03 00:06:30.135388 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.135361 2561 scope.go:117] "RemoveContainer" containerID="0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469" Jul 03 00:06:30.135604 ip-10-0-130-115 kubenswrapper[2561]: E0703 00:06:30.135585 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469\": container with ID starting with 0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469 not found: ID does not exist" containerID="0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469" Jul 03 00:06:30.135647 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.135610 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469"} err="failed to get container status \"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469\": rpc error: code = NotFound desc = could not find container \"0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469\": container with ID starting with 0414060aab2e89a69d14fec896d62090e9fc544f0e3339a1c303eeba0ebc3469 not found: ID does not exist" Jul 03 00:06:30.137544 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.137509 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 03 00:06:30.140795 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.140774 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c1893b-openshift-default-59467d8579-dzcxx"] Jul 03 00:06:30.446054 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.445972 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" path="/var/lib/kubelet/pods/ea23c676-d4d7-4dde-9627-671334217ee4/volumes" Jul 03 00:06:30.752828 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.752730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:30.853330 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:30.853272 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:31.028050 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:31.028015 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 03 00:06:31.028266 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:31.028243 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" containerID="cri-o://6e4b14d413bc328c931c19b93fa4922ca49e83a049b1588c4d413f1ecd1898b3" gracePeriod=30 Jul 03 00:06:31.042915 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:31.042890 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:31.306642 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:31.306561 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:36.152825 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.152797 2561 generic.go:356] "Generic (PLEG): container finished" podID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerID="6e4b14d413bc328c931c19b93fa4922ca49e83a049b1588c4d413f1ecd1898b3" exitCode=0 Jul 03 00:06:36.153179 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.152869 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerDied","Data":"6e4b14d413bc328c931c19b93fa4922ca49e83a049b1588c4d413f1ecd1898b3"} Jul 03 00:06:36.153179 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.152961 2561 scope.go:117] "RemoveContainer" containerID="29a461b9cfae7b302306c93203fed8f4608699258a08b090c54e878ae6fb1be4" Jul 03 00:06:36.290081 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.290052 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 03 00:06:36.369333 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369211 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmnvb\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369333 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369247 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369333 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369308 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369333 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369335 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369674 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369358 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369674 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369424 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369674 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369448 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369846 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369699 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:36.369846 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369773 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.369846 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369781 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 00:06:36.370009 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.369903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs\") pod \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\" (UID: \"2337058d-78c8-41ea-bac0-e1d768e5fcf5\") " Jul 03 00:06:36.370069 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.370016 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:36.370482 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.370459 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istiod-ca-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.370573 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.370487 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.370573 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.370483 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:36.370690 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.370603 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:36.371870 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.371842 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 00:06:36.371870 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.371849 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb" (OuterVolumeSpecName: "kube-api-access-wmnvb") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "kube-api-access-wmnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 00:06:36.372016 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.371864 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 00:06:36.372016 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.371854 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2337058d-78c8-41ea-bac0-e1d768e5fcf5" (UID: "2337058d-78c8-41ea-bac0-e1d768e5fcf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 00:06:36.471520 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471495 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-credential-socket\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471520 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471520 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-podinfo\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471659 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471531 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-token\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471659 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471539 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-data\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471659 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471547 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-istio-envoy\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471659 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471554 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2337058d-78c8-41ea-bac0-e1d768e5fcf5-workload-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.471659 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.471561 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnvb\" (UniqueName: \"kubernetes.io/projected/2337058d-78c8-41ea-bac0-e1d768e5fcf5-kube-api-access-wmnvb\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\"" Jul 03 00:06:36.848555 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.848524 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-758bb66cf4-78d6f_c21ee108-a2df-4902-8ac4-60d41eac1a0a/maas-api/0.log" Jul 03 00:06:36.963705 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:36.963662 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717265-hjxs2_9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c/cleanup/0.log" Jul 03 00:06:37.159627 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.159533 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" event={"ID":"2337058d-78c8-41ea-bac0-e1d768e5fcf5","Type":"ContainerDied","Data":"969aca8e2c0e20b343d8bd874911a96302b26d5fba9cddcb90efd34f992ba59e"} Jul 03 00:06:37.159627 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.159562 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf" Jul 03 00:06:37.159627 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.159604 2561 scope.go:117] "RemoveContainer" containerID="6e4b14d413bc328c931c19b93fa4922ca49e83a049b1588c4d413f1ecd1898b3" Jul 03 00:06:37.176715 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.176688 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 03 00:06:37.180521 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.180496 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c04a64-openshift-default-7fdbc8b95-5ctjf"] Jul 03 00:06:37.220854 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.220825 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cddbb748b-hpcrh_6179bbdb-6402-40a9-b296-68edeb54ef69/manager/0.log" Jul 03 00:06:37.344894 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.344864 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-szhhz_606f8fa2-c606-43cf-a864-93775055fc32/server/0.log" Jul 03 00:06:37.468297 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.468203 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hqbw7_0853faf4-ef79-430b-b4b8-fb0ddfa95883/manager/0.log" Jul 03 00:06:37.712784 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.712757 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f45646c76-9vpxj_73066ccd-4d2a-4aea-8dae-8a1d88ffff8e/copy-manifests/0.log" Jul 03 00:06:37.725427 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.725349 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f45646c76-9vpxj_73066ccd-4d2a-4aea-8dae-8a1d88ffff8e/manager/0.log" Jul 03 00:06:37.995511 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:37.995433 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-46c9c_d9ad423f-a519-44aa-8efb-90110de60c6d/postgres/0.log" Jul 03 00:06:38.447943 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.447912 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" path="/var/lib/kubelet/pods/2337058d-78c8-41ea-bac0-e1d768e5fcf5/volumes" Jul 03 00:06:38.806239 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.806214 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/util/0.log" Jul 03 00:06:38.812255 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.812232 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/pull/0.log" Jul 03 00:06:38.818089 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.818073 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/extract/0.log" Jul 03 00:06:38.939038 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.939008 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/extract/0.log" Jul 03 00:06:38.944662 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.944639 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/util/0.log" Jul 03 00:06:38.950361 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:38.950338 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/pull/0.log" Jul 03 00:06:39.067899 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.067826 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/util/0.log" Jul 03 00:06:39.073802 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.073783 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/pull/0.log" Jul 03 00:06:39.079836 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.079820 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/extract/0.log" Jul 03 00:06:39.194988 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.194956 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/extract/0.log" Jul 03 00:06:39.200730 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.200706 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/util/0.log" Jul 03 00:06:39.206358 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.206335 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/pull/0.log" Jul 03 00:06:39.569220 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.569188 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2nw4d_109e0945-51a7-46fa-abbe-af71229f0aea/manager/0.log" Jul 03 00:06:39.802063 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.802030 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bwmhb_c77bc0e8-3c14-4a05-b308-6437bd758eea/registry-server/0.log" Jul 03 00:06:39.924513 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:39.924441 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7scdw_da288574-77eb-4bd1-bf0c-9a083b54b2ba/manager/0.log" Jul 03 00:06:40.168106 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:40.168076 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gc44x_6737be2e-c7aa-42ff-b145-bc524e6ae53b/manager/0.log" Jul 03 00:06:40.660917 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:40.660886 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9jfct_c8c37a13-c7e3-426b-8c57-1f83d5d70ec3/discovery/0.log" Jul 03 00:06:40.891769 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:40.891742 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b444b4-hm9tw_ec117db5-3707-4617-b5e7-feea5fcc3ddd/kube-auth-proxy/0.log" Jul 03 00:06:41.132898 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:41.132861 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-tzvtn_c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff/payload-pre-processing/0.log" Jul 03 00:06:41.386857 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:41.386776 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9cdf7f67-8pwfb_60558dee-31d1-49d1-85ef-6ad6f53e142f/router/0.log" Jul 03 00:06:41.869106 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:41.869076 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pshls_217b86b7-f271-4643-8cdb-b883dafc66a1/main/0.log" Jul 03 00:06:42.130061 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:42.129983 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbcv6cb_a64abc7f-584c-4318-8953-8a22a57740a9/main/0.log" Jul 03 00:06:47.375617 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:47.375584 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:47.476982 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:47.476955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" limit=1024 Jul 03 00:06:50.195344 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:50.195310 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rx9mk_3729cf5d-1ad7-4025-bc82-9b93a24d1a44/global-pull-secret-syncer/0.log" Jul 03 00:06:50.240650 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:50.240621 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dpjvf_c2754126-6ebb-4f43-8ede-c67a663747a9/konnectivity-agent/0.log" Jul 03 00:06:50.340242 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:50.340210 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-115.ec2.internal_4b46bf19a4c8c74cc56a4f755de60986/haproxy/0.log" Jul 03 00:06:55.113873 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.113845 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/extract/0.log" Jul 03 00:06:55.137222 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.137200 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/util/0.log" Jul 03 00:06:55.156951 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.156929 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rpz8d_1f63806f-140b-41c9-bf27-c5d72c038aa3/pull/0.log" Jul 03 00:06:55.197793 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.197767 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/extract/0.log" Jul 03 00:06:55.219557 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.219537 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/util/0.log" Jul 03 00:06:55.244005 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.243963 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fjggh_92e471fe-1eb4-4c7f-9641-1a8b36f7b8ee/pull/0.log" Jul 03 00:06:55.277968 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.277949 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/extract/0.log" Jul 03 00:06:55.308547 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.308516 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/util/0.log" Jul 03 00:06:55.328691 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.328664 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jvxv2_abc67af1-61af-4328-b581-e93c82ac1ace/pull/0.log" Jul 03 00:06:55.353261 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.353232 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/extract/0.log" Jul 03 00:06:55.373972 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.373906 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/util/0.log" Jul 03 00:06:55.393244 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.393223 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpwbn_4f11ca3c-5950-404f-a30c-45c8177a743d/pull/0.log" Jul 03 00:06:55.931296 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:55.931250 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2nw4d_109e0945-51a7-46fa-abbe-af71229f0aea/manager/0.log" Jul 03 00:06:56.002579 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:56.002550 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bwmhb_c77bc0e8-3c14-4a05-b308-6437bd758eea/registry-server/0.log" Jul 03 00:06:56.091569 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:56.091541 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7scdw_da288574-77eb-4bd1-bf0c-9a083b54b2ba/manager/0.log" Jul 03 00:06:56.179469 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:56.179437 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gc44x_6737be2e-c7aa-42ff-b145-bc524e6ae53b/manager/0.log" Jul 03 00:06:58.060529 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.060497 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-cjjvx_6162850d-acc9-41e3-83f0-2c324b0e9f94/cluster-monitoring-operator/0.log" Jul 03 00:06:58.285004 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.284973 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cb85_5c3605bf-59df-464d-80eb-2fca1bc62813/node-exporter/0.log" Jul 03 00:06:58.309202 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.309177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cb85_5c3605bf-59df-464d-80eb-2fca1bc62813/kube-rbac-proxy/0.log" Jul 03 00:06:58.341584 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.341524 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cb85_5c3605bf-59df-464d-80eb-2fca1bc62813/init-textfile/0.log" Jul 03 00:06:58.427590 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.427558 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cg79l_0d0cc391-82c4-4c82-80dc-8dadee4a4a78/kube-rbac-proxy-main/0.log" Jul 03 00:06:58.447007 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.446983 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cg79l_0d0cc391-82c4-4c82-80dc-8dadee4a4a78/kube-rbac-proxy-self/0.log" Jul 03 00:06:58.466815 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.466791 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cg79l_0d0cc391-82c4-4c82-80dc-8dadee4a4a78/openshift-state-metrics/0.log" Jul 03 00:06:58.647188 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.647116 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-clm6t_8145dce9-24e4-4c48-a2c7-90d4c55e489b/prometheus-operator/0.log" Jul 03 00:06:58.668126 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:58.668104 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-clm6t_8145dce9-24e4-4c48-a2c7-90d4c55e489b/kube-rbac-proxy/0.log" Jul 03 00:06:59.887622 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.887582 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7"] Jul 03 00:06:59.888193 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888172 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888196 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888216 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888224 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888236 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888246 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888256 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.888303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888265 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.888647 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888427 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2337058d-78c8-41ea-bac0-e1d768e5fcf5" containerName="istio-proxy" Jul 03 00:06:59.888647 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888448 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.888647 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.888458 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea23c676-d4d7-4dde-9627-671334217ee4" containerName="istio-proxy" Jul 03 00:06:59.893230 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.893208 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:06:59.895602 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.895580 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x5drt\"/\"openshift-service-ca.crt\"" Jul 03 00:06:59.896505 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.896475 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x5drt\"/\"default-dockercfg-cjp6x\"" Jul 03 00:06:59.896505 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.896495 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x5drt\"/\"kube-root-ca.crt\"" Jul 03 00:06:59.901586 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.901558 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7"] Jul 03 00:06:59.986558 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.986514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxscx\" (UniqueName: \"kubernetes.io/projected/59cb614c-d280-4086-840a-d99b8edc1bf8-kube-api-access-vxscx\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:06:59.986558 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.986558 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-proc\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:06:59.986788 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.986600 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-lib-modules\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:06:59.986788 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.986624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-podres\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:06:59.986788 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:06:59.986709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-sys\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.087982 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.087946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxscx\" (UniqueName: \"kubernetes.io/projected/59cb614c-d280-4086-840a-d99b8edc1bf8-kube-api-access-vxscx\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.087982 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.087987 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-proc\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-lib-modules\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-podres\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-sys\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088097 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-proc\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088155 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-lib-modules\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088165 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-sys\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.088365 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.088216 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59cb614c-d280-4086-840a-d99b8edc1bf8-podres\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.095341 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.095319 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxscx\" (UniqueName: \"kubernetes.io/projected/59cb614c-d280-4086-840a-d99b8edc1bf8-kube-api-access-vxscx\") pod \"perf-node-gather-daemonset-hjgn7\" (UID: \"59cb614c-d280-4086-840a-d99b8edc1bf8\") " pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.205542 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.205454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:00.349982 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.349951 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7"] Jul 03 00:07:00.351296 ip-10-0-130-115 kubenswrapper[2561]: W0703 00:07:00.351254 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59cb614c_d280_4086_840a_d99b8edc1bf8.slice/crio-0dcbcc19db47a5f4155cb1bae453480c0ff21fb29dc664c82991a2355db89601 WatchSource:0}: Error finding container 0dcbcc19db47a5f4155cb1bae453480c0ff21fb29dc664c82991a2355db89601: Status 404 returned error can't find the container with id 0dcbcc19db47a5f4155cb1bae453480c0ff21fb29dc664c82991a2355db89601 Jul 03 00:07:00.580160 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.580133 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/2.log" Jul 03 00:07:00.583964 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.583944 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hjwk8_ab3548bc-f22e-4b6e-a727-a342457af286/console-operator/3.log" Jul 03 00:07:00.985082 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:00.985056 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cf7d699f9-gq8xp_751e6bd0-5c22-44eb-bf6f-77708fd8fbab/console/0.log" Jul 03 00:07:01.300001 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:01.299968 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" event={"ID":"59cb614c-d280-4086-840a-d99b8edc1bf8","Type":"ContainerStarted","Data":"5a57a8363b5ada673b793ae38e368dd44a3dd6c9446c2d379fcf06230ea874d4"} Jul 03 00:07:01.300165 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:01.300020 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" event={"ID":"59cb614c-d280-4086-840a-d99b8edc1bf8","Type":"ContainerStarted","Data":"0dcbcc19db47a5f4155cb1bae453480c0ff21fb29dc664c82991a2355db89601"} Jul 03 00:07:01.300165 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:01.300100 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:01.315631 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:01.315582 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" podStartSLOduration=2.315568855 podStartE2EDuration="2.315568855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 00:07:01.313406292 +0000 UTC m=+2675.395730217" watchObservedRunningTime="2026-07-03 00:07:01.315568855 +0000 UTC m=+2675.397892779" Jul 03 00:07:01.557371 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:01.557274 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-x8bpt_09c6f89b-4757-4174-ace6-96c36a9e0706/volume-data-source-validator/0.log" Jul 03 00:07:02.573181 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:02.573137 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s5tnc_25b4864a-b323-4420-a61c-86f098df9474/dns/0.log" Jul 03 00:07:02.591532 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:02.591507 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s5tnc_25b4864a-b323-4420-a61c-86f098df9474/kube-rbac-proxy/0.log" Jul 03 00:07:02.632590 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:02.632561 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pq92b_119d9db1-dbe0-4358-a3bb-315aac8af83a/dns-node-resolver/0.log" Jul 03 00:07:03.218147 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:03.218106 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2hx8_e2091d71-7dcc-43ad-807d-3ded4bf54cec/node-ca/0.log" Jul 03 00:07:04.217440 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:04.217412 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9jfct_c8c37a13-c7e3-426b-8c57-1f83d5d70ec3/discovery/0.log" Jul 03 00:07:04.261345 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:04.261314 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b444b4-hm9tw_ec117db5-3707-4617-b5e7-feea5fcc3ddd/kube-auth-proxy/0.log" Jul 03 00:07:04.379192 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:04.379164 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-tzvtn_c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff/payload-pre-processing/0.log" Jul 03 00:07:04.471850 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:04.471758 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9cdf7f67-8pwfb_60558dee-31d1-49d1-85ef-6ad6f53e142f/router/0.log" Jul 03 00:07:05.131897 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.131861 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2g5lx_03794c72-91ad-4ff1-87e5-16e6520e708e/serve-healthcheck-canary/0.log" Jul 03 00:07:05.602356 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.602317 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wjjdb_49a946af-8ab2-4774-a67e-5125b517005e/insights-operator/0.log" Jul 03 00:07:05.602878 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.602810 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wjjdb_49a946af-8ab2-4774-a67e-5125b517005e/insights-operator/1.log" Jul 03 00:07:05.755077 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.755046 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klgwl_b18286e6-f2b7-48f6-95bb-e52b53812325/kube-rbac-proxy/0.log" Jul 03 00:07:05.773059 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.773021 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klgwl_b18286e6-f2b7-48f6-95bb-e52b53812325/exporter/0.log" Jul 03 00:07:05.791184 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:05.791162 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klgwl_b18286e6-f2b7-48f6-95bb-e52b53812325/extractor/0.log" Jul 03 00:07:07.315992 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:07.315961 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x5drt/perf-node-gather-daemonset-hjgn7" Jul 03 00:07:08.017635 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.017604 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-758bb66cf4-78d6f_c21ee108-a2df-4902-8ac4-60d41eac1a0a/maas-api/0.log" Jul 03 00:07:08.036265 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.036239 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717265-hjxs2_9ae9e18e-81c4-4ca4-9bfa-70ac44043c5c/cleanup/0.log" Jul 03 00:07:08.168236 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.168193 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cddbb748b-hpcrh_6179bbdb-6402-40a9-b296-68edeb54ef69/manager/0.log" Jul 03 00:07:08.189025 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.188991 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-szhhz_606f8fa2-c606-43cf-a864-93775055fc32/server/0.log" Jul 03 00:07:08.252150 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.252103 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hqbw7_0853faf4-ef79-430b-b4b8-fb0ddfa95883/manager/0.log" Jul 03 00:07:08.365360 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.365264 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f45646c76-9vpxj_73066ccd-4d2a-4aea-8dae-8a1d88ffff8e/manager/0.log" Jul 03 00:07:08.381195 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.381164 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f45646c76-9vpxj_73066ccd-4d2a-4aea-8dae-8a1d88ffff8e/copy-manifests/0.log" Jul 03 00:07:08.460572 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:08.460549 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-46c9c_d9ad423f-a519-44aa-8efb-90110de60c6d/postgres/0.log" Jul 03 00:07:10.002531 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:10.002452 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9jfct_c8c37a13-c7e3-426b-8c57-1f83d5d70ec3/discovery/0.log" Jul 03 00:07:10.027434 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:10.027411 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b444b4-hm9tw_ec117db5-3707-4617-b5e7-feea5fcc3ddd/kube-auth-proxy/0.log" Jul 03 00:07:10.123210 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:10.123182 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-tzvtn_c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff/payload-pre-processing/0.log" Jul 03 00:07:10.209718 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:10.209673 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9cdf7f67-8pwfb_60558dee-31d1-49d1-85ef-6ad6f53e142f/router/0.log" Jul 03 00:07:11.472142 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:11.472109 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9jfct_c8c37a13-c7e3-426b-8c57-1f83d5d70ec3/discovery/0.log" Jul 03 00:07:11.496524 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:11.496500 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b444b4-hm9tw_ec117db5-3707-4617-b5e7-feea5fcc3ddd/kube-auth-proxy/0.log" Jul 03 00:07:11.596248 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:11.596217 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-tzvtn_c7cf40d6-c9a6-40dc-a9bc-ddbbd66641ff/payload-pre-processing/0.log" Jul 03 00:07:11.676688 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:11.676612 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9cdf7f67-8pwfb_60558dee-31d1-49d1-85ef-6ad6f53e142f/router/0.log" Jul 03 00:07:14.965303 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:14.965255 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-vtf6x_aa35770d-8b0e-42b8-bed0-b4f90a9eed3d/kube-storage-version-migrator-operator/1.log" Jul 03 00:07:14.966158 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:14.966139 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-vtf6x_aa35770d-8b0e-42b8-bed0-b4f90a9eed3d/kube-storage-version-migrator-operator/0.log" Jul 03 00:07:16.072234 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.072208 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/kube-multus-additional-cni-plugins/0.log" Jul 03 00:07:16.091102 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.091077 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/egress-router-binary-copy/0.log" Jul 03 00:07:16.110010 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.109982 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/cni-plugins/0.log" Jul 03 00:07:16.131379 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.131353 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/bond-cni-plugin/0.log" Jul 03 00:07:16.150921 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.150894 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/routeoverride-cni/0.log" Jul 03 00:07:16.169111 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.169087 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/whereabouts-cni-bincopy/0.log" Jul 03 00:07:16.186496 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.186473 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwsbr_e24fae4e-4711-4b22-bd1e-066b882107a3/whereabouts-cni/0.log" Jul 03 00:07:16.355541 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.355463 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fztcg_361985eb-e209-4f32-8465-45c2d2528cb3/kube-multus/0.log" Jul 03 00:07:16.408484 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.408450 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-59s58_412e9e17-1969-40a1-829b-79346d9d4d34/network-metrics-daemon/0.log" Jul 03 00:07:16.425155 ip-10-0-130-115 kubenswrapper[2561]: I0703 00:07:16.425134 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-59s58_412e9e17-1969-40a1-829b-79346d9d4d34/kube-rbac-proxy/0.log"