Jul 20 13:16:39.573269 ip-10-0-141-54 systemd[1]: Starting Kubernetes Kubelet... Jul 20 13:16:39.894271 ip-10-0-141-54 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:39.894271 ip-10-0-141-54 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 13:16:39.894271 ip-10-0-141-54 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:39.894271 ip-10-0-141-54 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 13:16:39.894271 ip-10-0-141-54 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:39.895564 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.895480 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 13:16:39.898765 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898750 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.898765 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898765 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898769 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898771 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898774 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898777 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898780 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898782 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898785 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898788 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898790 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898793 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898796 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898798 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898801 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898804 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898807 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898810 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898813 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898815 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898817 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.898825 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898820 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898824 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898826 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898829 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898832 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898835 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898838 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898840 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898843 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898845 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898848 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898850 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898853 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898855 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898858 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898860 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898863 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898865 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898868 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898871 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.899302 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898873 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898875 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898878 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898880 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898883 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898885 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898887 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898890 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898892 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898895 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898897 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898899 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898902 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898904 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898908 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898911 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898913 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898916 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898918 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898921 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.899868 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898923 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898926 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898928 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898931 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898933 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898936 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898938 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898941 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898943 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898946 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898949 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898951 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898954 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898956 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898970 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898973 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898975 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898980 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898983 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898986 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.900362 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898989 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898992 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898995 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.898998 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899001 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899004 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899007 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899010 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899013 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899017 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899021 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899024 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899027 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899030 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899032 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899035 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899039 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899042 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899703 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.900830 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899708 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899712 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899714 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899717 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899727 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899730 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899733 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899736 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899738 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899741 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899744 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899746 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899749 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899751 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899754 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899757 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899759 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899762 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899765 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.901287 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899771 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899775 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899778 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899781 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899784 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899787 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899790 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899792 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899795 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899798 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899801 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899803 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899806 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899809 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899813 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899816 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899819 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899823 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899827 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.901744 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899829 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899832 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899834 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899837 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899839 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899842 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899844 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899846 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899849 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899851 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899855 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899857 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899860 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899864 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899867 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899870 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899873 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899875 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899878 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899881 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.902214 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899883 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899886 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899888 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899891 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899893 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899896 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899898 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899901 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899904 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899907 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899910 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899912 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899915 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899917 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899920 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899922 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899925 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899928 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899930 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899932 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.902709 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899935 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899937 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899940 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899942 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899944 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899948 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899951 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899954 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899956 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899974 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899977 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899980 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899982 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899985 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899988 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899991 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899993 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899996 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.899998 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900001 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.903225 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900403 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900408 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900412 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900416 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900419 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900422 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900425 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900428 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900431 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900434 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900436 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900439 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900442 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900444 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900446 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900449 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900451 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900454 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900457 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.903730 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900460 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900462 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900465 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900468 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900470 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900473 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900476 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900478 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900481 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900483 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900486 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900488 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900491 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900493 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900496 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900499 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900502 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900504 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900507 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.904211 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900510 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900513 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900515 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900518 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900520 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900522 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900525 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900527 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900530 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900532 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900534 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900537 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900540 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900542 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900547 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900550 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900552 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900555 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900557 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900560 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.904662 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900562 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900565 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900568 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900570 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900573 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900575 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900578 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900580 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900583 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900588 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900590 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900593 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900596 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900598 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900601 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900604 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900606 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900609 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900611 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900614 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.905185 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900616 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900619 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900623 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900628 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900631 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900634 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900636 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900641 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900644 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900647 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900650 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900653 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900655 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900658 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900660 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900663 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900665 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900668 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900670 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.905658 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900673 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.900675 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900753 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900768 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900775 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900779 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900786 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900789 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900793 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900797 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900801 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900804 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900807 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900810 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900813 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900816 2577 flags.go:64] FLAG: --cgroup-root="" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900819 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900823 2577 flags.go:64] FLAG: --client-ca-file="" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900825 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900828 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900833 2577 flags.go:64] FLAG: --cluster-domain="" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900836 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900839 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900842 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 13:16:39.906164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900845 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900849 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900852 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900855 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900858 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900861 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900864 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900867 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900870 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900873 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900879 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900882 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900885 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900887 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900891 2577 flags.go:64] FLAG: --enable-server="true" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900894 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900899 2577 flags.go:64] FLAG: --event-burst="100" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900902 2577 flags.go:64] FLAG: --event-qps="50" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900905 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900908 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900910 2577 flags.go:64] FLAG: --eviction-hard="" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900914 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900917 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900920 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900923 2577 flags.go:64] FLAG: --eviction-soft="" Jul 20 13:16:39.906741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900926 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900929 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900932 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900935 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900937 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900940 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900943 2577 flags.go:64] FLAG: --feature-gates="" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900947 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900950 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900953 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900956 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900971 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900974 2577 flags.go:64] FLAG: --help="false" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900977 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-54.ec2.internal" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900980 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900983 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900986 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900991 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900994 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.900997 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901000 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901002 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901005 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 13:16:39.907392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901008 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901011 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901014 2577 flags.go:64] FLAG: --kube-reserved="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901017 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901020 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901023 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901025 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901028 2577 flags.go:64] FLAG: --lock-file="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901031 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901034 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901037 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901042 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901045 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901048 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901051 2577 flags.go:64] FLAG: --logging-format="text" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901053 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901057 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901059 2577 flags.go:64] FLAG: --manifest-url="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901062 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901066 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901070 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901073 2577 flags.go:64] FLAG: --max-pods="110" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901076 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901092 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901096 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 13:16:39.907979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901099 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901104 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901107 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901110 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901116 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901119 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901123 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901126 2577 flags.go:64] FLAG: --pod-cidr="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901129 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901134 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901137 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901141 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901144 2577 flags.go:64] FLAG: --port="10250" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901147 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901150 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e3c4aaa3313fea5" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901153 2577 flags.go:64] FLAG: --qos-reserved="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901156 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901159 2577 flags.go:64] FLAG: --register-node="true" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901162 2577 flags.go:64] FLAG: --register-with-taints="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901165 2577 flags.go:64] FLAG: --registry-burst="10" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901168 2577 flags.go:64] FLAG: --registry-qps="5" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901171 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901174 2577 flags.go:64] FLAG: --reserved-memory="" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901177 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901180 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 13:16:39.908582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901183 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901186 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901189 2577 flags.go:64] FLAG: --runonce="false" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901192 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901195 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901197 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901200 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901203 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901206 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901211 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901214 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901216 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901219 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901222 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901225 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901228 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901232 2577 flags.go:64] FLAG: --system-cgroups="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901234 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901239 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901242 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901245 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901254 2577 flags.go:64] FLAG: --tls-min-version="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901256 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901259 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901262 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 13:16:39.909203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901265 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901267 2577 flags.go:64] FLAG: --v="2" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901271 2577 flags.go:64] FLAG: --version="false" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901275 2577 flags.go:64] FLAG: --vmodule="" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901279 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.901282 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901393 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901397 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901401 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901403 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901406 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901409 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901411 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901414 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901416 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901419 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901422 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901425 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901428 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901431 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901433 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.909832 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901436 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901438 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901441 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901444 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901446 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901449 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901452 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901457 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901459 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901462 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901465 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901469 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901473 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901476 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901478 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901481 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901484 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901486 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901489 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901491 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.910378 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901494 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901496 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901499 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901502 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901504 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901507 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901509 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901513 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901516 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901518 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901521 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901523 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901526 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901529 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901531 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901534 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901536 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901539 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901542 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901546 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.910877 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901548 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901550 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901553 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901555 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901558 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901561 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901564 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901568 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901570 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901573 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901576 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901580 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901583 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901586 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901589 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901592 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901595 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901597 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901600 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.911374 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901604 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901607 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901609 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901612 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901614 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901617 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901619 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901622 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901625 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901627 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901630 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901633 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901637 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901639 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901642 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901644 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901646 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901649 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901651 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901654 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.911831 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901656 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901659 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901661 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901664 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.901666 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.902194 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.908569 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.908584 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.908596 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.908607 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908662 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908667 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908670 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908674 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.912335 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908679 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908682 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908685 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908688 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908690 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908693 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908696 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908699 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908701 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908704 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908706 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908709 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908711 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908714 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908717 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908719 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908722 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908724 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908727 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.912685 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908730 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908732 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908734 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908737 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908739 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908742 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908746 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908749 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908751 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908754 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908757 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908759 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908762 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908765 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908767 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908771 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908775 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908777 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908780 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.913160 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908783 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908786 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908788 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908791 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908794 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908796 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908799 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908801 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908804 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908806 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908809 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908812 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908814 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908816 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908819 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908822 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908824 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908827 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908830 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908833 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.913610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908836 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908838 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908841 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908843 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908846 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908849 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908851 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908854 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908856 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908858 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908861 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908863 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908866 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908868 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908871 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908873 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908877 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908879 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908882 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908885 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.914113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908887 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908890 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908892 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908895 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908897 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908900 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908902 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908906 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908908 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908912 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908916 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908919 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908922 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908925 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908928 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908931 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.914606 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.908934 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.908939 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909064 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909069 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909073 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909075 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909078 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909081 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909084 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909087 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909089 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909092 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909095 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909097 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909100 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:39.915183 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909102 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909105 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909107 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909110 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909112 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909115 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909117 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909120 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909122 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909125 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909127 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909130 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909132 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909135 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909138 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909141 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909143 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909146 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909148 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:39.915787 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909151 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909155 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909157 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909160 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909163 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909166 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909168 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909170 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909173 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909175 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909178 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909180 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909183 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909185 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909188 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909190 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909192 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909195 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909197 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909200 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:39.916514 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909202 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909205 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909208 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909211 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909214 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909218 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909222 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909224 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909227 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909231 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909234 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909237 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909240 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909242 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909245 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909247 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909250 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909253 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909255 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:39.917056 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909257 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909260 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909263 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909265 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909268 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909270 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909273 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909275 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909278 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909280 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909283 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909285 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909287 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909290 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909292 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909295 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909297 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909300 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909303 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909305 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:39.917511 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909308 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909310 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909313 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909320 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909324 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909327 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909330 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:39.909333 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.909338 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.909457 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.911605 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.912799 2577 server.go:1019] "Starting client certificate rotation" Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.912896 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:39.918111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.912955 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:39.918434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.916883 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:39.919434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.919417 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:39.929409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.929389 2577 log.go:25] "Validated CRI v1 runtime API" Jul 20 13:16:39.933433 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.933415 2577 log.go:25] "Validated CRI v1 image API" Jul 20 13:16:39.934266 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.934248 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 13:16:39.936244 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.936226 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e46aafd3-f894-4593-85de-05e8549dd976:/dev/nvme0n1p3 fe6b1a31-4fb3-41ac-94e3-3cb2286a5ae1:/dev/nvme0n1p4] Jul 20 13:16:39.936299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.936244 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 13:16:39.941085 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.940978 2577 manager.go:216] Machine: {Timestamp:2026-07-20 13:16:39.93985481 +0000 UTC m=+0.268412195 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099579 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227fb200098b9e9f5079dfb9cd8bdd SystemUUID:ec227fb2-0009-8b9e-9f50-79dfb9cd8bdd BootID:6810b68b-0964-4055-8ee6-b3d08f3cc2df Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:63:57:6b:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:63:57:6b:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:9a:4c:f4:05:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 13:16:39.941085 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.941083 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 13:16:39.941196 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:39.941185 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 13:16:41.207809 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.207785 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 13:16:41.207969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.207814 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-54.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 13:16:41.208491 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.208481 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 13:16:41.208523 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.208495 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 13:16:41.208523 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.208503 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:41.209054 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.209043 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:41.210203 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.210192 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 13:16:41.213934 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.213922 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:41.214076 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214067 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 13:16:41.214653 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214644 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 20 13:16:41.214684 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214658 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 13:16:41.214684 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214673 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 13:16:41.214684 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214683 2577 kubelet.go:398] "Adding apiserver pod source" Jul 20 13:16:41.214815 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.214691 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 13:16:41.215041 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.215030 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:41.215077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.215048 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:41.216630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.216613 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 13:16:41.219491 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.219464 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 13:16:41.219586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.219554 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 13:16:41.221465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221449 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221470 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221479 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221488 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221501 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221510 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221521 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221529 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221539 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221547 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 13:16:41.221559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221560 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 13:16:41.221834 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.221568 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 13:16:41.222200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.222188 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 13:16:41.222274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.222203 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 13:16:41.222274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.222251 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:41.225448 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.225432 2577 server.go:1268] "Started kubelet" Jul 20 13:16:41.225524 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.225498 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 13:16:41.225807 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.225747 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 13:16:41.225881 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.225812 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 13:16:41.226233 ip-10-0-141-54 systemd[1]: Started Kubernetes Kubelet. Jul 20 13:16:41.227387 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.227300 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 13:16:41.227488 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.227444 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 20 13:16:41.229197 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.229155 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-54.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 13:16:41.229299 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.229241 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 13:16:41.229359 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.229342 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 13:16:41.230004 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.229238 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0ac16d85 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.225407877 +0000 UTC m=+0.289756783,LastTimestamp:2026-07-20 13:16:41.225407877 +0000 UTC m=+0.289756783,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.230542 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.230497 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 13:16:41.234313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234298 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 13:16:41.234373 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234309 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:41.234500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.234474 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.234862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234840 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 13:16:41.234862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234842 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 13:16:41.234992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234868 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 13:16:41.234992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234966 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 20 13:16:41.234992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.234972 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 20 13:16:41.235125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235074 2577 factory.go:55] Registering systemd factory Jul 20 13:16:41.235125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235089 2577 factory.go:223] Registration of the systemd container factory successfully Jul 20 13:16:41.235494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235476 2577 factory.go:153] Registering CRI-O factory Jul 20 13:16:41.235494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235495 2577 factory.go:223] Registration of the crio container factory successfully Jul 20 13:16:41.235619 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235549 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 13:16:41.235619 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235574 2577 factory.go:103] Registering Raw factory Jul 20 13:16:41.235619 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.235590 2577 manager.go:1196] Started watching for new ooms in manager Jul 20 13:16:41.236561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.236478 2577 manager.go:349] Starting recovery of all containers Jul 20 13:16:41.238743 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.238716 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 13:16:41.239231 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.239205 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 13:16:41.247942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.247924 2577 manager.go:354] Recovery completed Jul 20 13:16:41.252643 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.252629 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.254428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254409 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.254502 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254442 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.254502 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254452 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.254899 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254880 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 13:16:41.254899 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254897 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 13:16:41.255024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254906 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.255024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.254913 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:41.255605 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.255594 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.255678 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.255616 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.255678 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.255626 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.257057 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.257040 2577 policy_none.go:49] "None policy: Start" Jul 20 13:16:41.257057 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.257056 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 13:16:41.257269 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.257072 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 13:16:41.259124 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.259111 2577 policy_none.go:47] "Start" Jul 20 13:16:41.269484 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.269415 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7c3b7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254427517 +0000 UTC m=+0.318776428,LastTimestamp:2026-07-20 13:16:41.254427517 +0000 UTC m=+0.318776428,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.278550 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.278449 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7c86ea default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254446826 +0000 UTC m=+0.318795732,LastTimestamp:2026-07-20 13:16:41.254446826 +0000 UTC m=+0.318795732,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.286659 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.286598 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7cab27 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254456103 +0000 UTC m=+0.318805009,LastTimestamp:2026-07-20 13:16:41.254456103 +0000 UTC m=+0.318805009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.296709 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.296697 2577 manager.go:337] "Starting Device Plugin manager" Jul 20 13:16:41.296774 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.296726 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 13:16:41.296774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.296737 2577 server.go:87] "Starting device plugin registration server" Jul 20 13:16:41.296953 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.296941 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 13:16:41.297050 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.296957 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 13:16:41.297090 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.297059 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 13:16:41.297449 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.297129 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 13:16:41.297449 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.297137 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 13:16:41.297717 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.297696 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 13:16:41.297778 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.297753 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.300953 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.300892 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-54.ec2.internal.18c401dd0c7c3b7d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7c3b7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254427517 +0000 UTC m=+0.318776428,LastTimestamp:2026-07-20 13:16:41.255604558 +0000 UTC m=+0.319953464,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.311922 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.311862 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-54.ec2.internal.18c401dd0c7c86ea\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7c86ea default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254446826 +0000 UTC m=+0.318795732,LastTimestamp:2026-07-20 13:16:41.255621065 +0000 UTC m=+0.319969971,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.335341 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.312349 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99qrf" Jul 20 13:16:41.335341 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.321419 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99qrf" Jul 20 13:16:41.335341 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.322568 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-54.ec2.internal.18c401dd0c7cab27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-54.ec2.internal.18c401dd0c7cab27 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-54.ec2.internal,UID:ip-10-0-141-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-54.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-54.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.254456103 +0000 UTC m=+0.318805009,LastTimestamp:2026-07-20 13:16:41.255629391 +0000 UTC m=+0.319978299,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-54.ec2.internal,}" Jul 20 13:16:41.384976 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.384935 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 13:16:41.386153 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.386137 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 13:16:41.386235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.386190 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 13:16:41.386235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.386214 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 13:16:41.386315 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.386248 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 13:16:41.388893 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.388877 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:41.398070 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.398056 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.398796 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.398782 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.398852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.398811 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.398852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.398828 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.398852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.398851 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.408119 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.408105 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.408186 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.408126 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-54.ec2.internal\": node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.450150 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.450050 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.487367 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.487341 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal"] Jul 20 13:16:41.487486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.487405 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.488852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.488835 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.488980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.488870 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.488980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.488884 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.491298 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.491282 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.491401 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.491386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.491449 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.491416 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.492569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492550 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.492630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492577 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.492630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492587 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.492630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492559 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.492769 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492651 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.492769 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.492667 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.494849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.494832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.494931 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.494864 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.495472 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.495459 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.495556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.495483 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.495556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.495497 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.519767 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.519745 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-54.ec2.internal\" not found" node="ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.524274 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.524258 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-54.ec2.internal\" not found" node="ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.536919 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.536897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.536980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.536925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.536980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.536942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73f6bdbf997a544246c875c3b1e88545-config\") pod \"kube-apiserver-proxy-ip-10-0-141-54.ec2.internal\" (UID: \"73f6bdbf997a544246c875c3b1e88545\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.551012 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.550983 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.638073 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.638073 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.638255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73f6bdbf997a544246c875c3b1e88545-config\") pod \"kube-apiserver-proxy-ip-10-0-141-54.ec2.internal\" (UID: \"73f6bdbf997a544246c875c3b1e88545\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.638255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638143 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.638255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a498e25e8920f5d6324995f0e2bde44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal\" (UID: \"0a498e25e8920f5d6324995f0e2bde44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.638255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.638144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73f6bdbf997a544246c875c3b1e88545-config\") pod \"kube-apiserver-proxy-ip-10-0-141-54.ec2.internal\" (UID: \"73f6bdbf997a544246c875c3b1e88545\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.651833 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.651811 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.752744 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.752691 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.822910 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.822880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.826674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:41.826654 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:41.853875 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.853686 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:41.954207 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:41.954153 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.054662 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.054590 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.155180 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.155151 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.176327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.176293 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 13:16:42.176721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.176463 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 13:16:42.255447 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.255421 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.326819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.326729 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 13:11:41 +0000 UTC" deadline="2027-12-15 23:32:38.128815005 +0000 UTC" Jul 20 13:16:42.326819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.326772 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12322h15m55.802047084s" Jul 20 13:16:42.356228 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.356197 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.380358 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.380335 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:42.381284 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.381267 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:42.381354 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.381300 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:42.381354 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.381311 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:42.443627 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.443604 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:42.456447 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.456424 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.557085 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:42.557055 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-54.ec2.internal\" not found" Jul 20 13:16:42.590947 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:42.590915 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a498e25e8920f5d6324995f0e2bde44.slice/crio-c7571eba95a2f582fab52efb6aacd3d809dd43e52edabc653acc6d19f93eb1dd WatchSource:0}: Error finding container c7571eba95a2f582fab52efb6aacd3d809dd43e52edabc653acc6d19f93eb1dd: Status 404 returned error can't find the container with id c7571eba95a2f582fab52efb6aacd3d809dd43e52edabc653acc6d19f93eb1dd Jul 20 13:16:42.591204 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:42.591157 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f6bdbf997a544246c875c3b1e88545.slice/crio-030c0e329a289faae738204a0d91c949707e948a0a3b9c0973a9402f627e37d1 WatchSource:0}: Error finding container 030c0e329a289faae738204a0d91c949707e948a0a3b9c0973a9402f627e37d1: Status 404 returned error can't find the container with id 030c0e329a289faae738204a0d91c949707e948a0a3b9c0973a9402f627e37d1 Jul 20 13:16:42.595823 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.595807 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:16:42.609843 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.609823 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:42.635028 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.635001 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" Jul 20 13:16:42.650791 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.649473 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:42.651916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.651881 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" Jul 20 13:16:42.661853 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.661838 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:42.674414 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:42.674388 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:43.216202 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.216163 2577 apiserver.go:52] "Watching apiserver" Jul 20 13:16:43.224466 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.224432 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 13:16:43.224878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.224844 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-r5s2p","openshift-image-registry/node-ca-9kkq8","openshift-multus/multus-additional-cni-plugins-hl96x","openshift-multus/multus-lx8xz","openshift-network-operator/iptables-alerter-56xc8","openshift-ovn-kubernetes/ovnkube-node-hddgc","kube-system/global-pull-secret-syncer-mtlhw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5","openshift-dns/node-resolver-mxjhc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal","openshift-multus/network-metrics-daemon-52sdt","openshift-network-diagnostics/network-check-target-txwcc","kube-system/konnectivity-agent-5wbzb","kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal"] Jul 20 13:16:43.227407 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.227385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.227505 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.227466 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:43.230138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.230119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.232462 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.232437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.234586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.234560 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:43.235125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.234942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.237494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.237462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.239785 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.239766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.242051 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.242033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.244376 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.244357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.245501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245483 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.245609 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245583 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfznd\"" Jul 20 13:16:43.245681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245673 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.245754 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8bpdd\"" Jul 20 13:16:43.245877 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245860 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.245877 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.245873 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjmd5\"" Jul 20 13:16:43.247050 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247032 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.247138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-bin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsprh\" (UniqueName: \"kubernetes.io/projected/01e9b533-764f-4eef-a8b1-80dc2067036a-kube-api-access-jsprh\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-cnibin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-cni-binary-copy\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-k8s-cni-cncf-io\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-conf-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsl4j\" (UniqueName: \"kubernetes.io/projected/fffe4b43-27dc-4eac-b530-675d9773da6f-kube-api-access-hsl4j\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-os-release\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-kubelet\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-hostroot\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-multus-certs\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-dbus\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.247778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e949be8-73ac-4b83-b1b6-0a1ace341746-konnectivity-ca\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.247778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-system-cni-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-cnibin\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-socket-dir-parent\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-os-release\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-netns\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-daemon-config\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.247975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eef561c0-bb8e-48e0-8d2d-d9357819fe85-iptables-alerter-script\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.247994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kpp\" (UniqueName: \"kubernetes.io/projected/eef561c0-bb8e-48e0-8d2d-d9357819fe85-kube-api-access-z8kpp\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-kubelet-config\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e949be8-73ac-4b83-b1b6-0a1ace341746-agent-certs\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-system-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-etc-kubernetes\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef561c0-bb8e-48e0-8d2d-d9357819fe85-host-slash\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.248272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.248642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.248642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.248337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-multus\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.249311 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.249290 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 13:16:43.249611 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.249592 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kskvr\"" Jul 20 13:16:43.249719 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.249661 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sq28r\"" Jul 20 13:16:43.250828 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.250795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.252558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.252522 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.252844 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.252826 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 13:16:43.253083 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.253063 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.253382 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.253364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 13:16:43.253739 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.253723 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 13:16:43.254256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.254037 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 13:16:43.254256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.254057 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 13:16:43.254256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.254085 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.254256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.254146 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.254811 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.254791 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255021 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255241 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255320 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlwwv\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255399 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255533 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.255558 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.256015 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.256099 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 13:16:43.256257 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.256267 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 13:16:43.256997 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.256949 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8h8gr\"" Jul 20 13:16:43.258658 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.257993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.258658 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.258078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:43.259436 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.259418 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 13:16:43.259579 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.259447 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 13:16:43.259670 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.259592 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 13:16:43.260245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.259961 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-slbzm\"" Jul 20 13:16:43.260245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.260073 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tznzm\"" Jul 20 13:16:43.260936 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.260922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:43.261016 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.260983 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:43.273800 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.273637 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:43.295878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.295852 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8mnk" Jul 20 13:16:43.297013 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.296993 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:43.303822 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.303803 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8mnk" Jul 20 13:16:43.336676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.336653 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 13:16:43.349490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-bin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-log-socket\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d220c08-dc87-46d4-a868-b10a521c3b40-hosts-file\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-conf\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-bin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsprh\" (UniqueName: \"kubernetes.io/projected/01e9b533-764f-4eef-a8b1-80dc2067036a-kube-api-access-jsprh\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-cnibin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.349635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-k8s-cni-cncf-io\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsl4j\" (UniqueName: \"kubernetes.io/projected/fffe4b43-27dc-4eac-b530-675d9773da6f-kube-api-access-hsl4j\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-netd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-kubernetes\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-k8s-cni-cncf-io\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349717 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-cnibin\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-os-release\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-script-lib\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mqk\" (UniqueName: \"kubernetes.io/projected/5daed80f-e228-440a-aed0-8c423a2044bf-kube-api-access-82mqk\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-hostroot\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-multus-certs\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-os-release\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-hostroot\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.349959 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-kubelet\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-multus-certs\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjft\" (UniqueName: \"kubernetes.io/projected/8dec058e-f451-44de-a6a7-46f9725fd4c8-kube-api-access-ckjft\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c815171-5cd5-4db5-b95a-dc9395fb6000-host\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/9c815171-5cd5-4db5-b95a-dc9395fb6000-kube-api-access-n79mn\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-dbus\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-socket-dir-parent\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-node-log\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d220c08-dc87-46d4-a868-b10a521c3b40-tmp-dir\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysconfig\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-tuned\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-netns\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-socket-dir-parent\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eef561c0-bb8e-48e0-8d2d-d9357819fe85-iptables-alerter-script\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-run-netns\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-dbus\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kpp\" (UniqueName: \"kubernetes.io/projected/eef561c0-bb8e-48e0-8d2d-d9357819fe85-kube-api-access-z8kpp\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-systemd-units\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.350764 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpm7\" (UniqueName: \"kubernetes.io/projected/11833f47-b5c7-4ad0-abf8-878cde93e3c9-kube-api-access-nqpm7\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-kubelet-config\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e949be8-73ac-4b83-b1b6-0a1ace341746-agent-certs\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-system-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef561c0-bb8e-48e0-8d2d-d9357819fe85-host-slash\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-multus\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-etc-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed66974-03a7-40cc-9cea-65fc2a99cd57-kubelet-config\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-registration-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-config\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-sys\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-system-cni-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.351561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-lib-modules\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef561c0-bb8e-48e0-8d2d-d9357819fe85-host-slash\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-tmp\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnmc\" (UniqueName: \"kubernetes.io/projected/947acb8b-57e8-4622-a44a-0b33e8c4a078-kube-api-access-jxnmc\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-socket-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-cni-binary-copy\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-conf-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.350964 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eef561c0-bb8e-48e0-8d2d-d9357819fe85-iptables-alerter-script\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovn-node-metrics-cert\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-systemd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-env-overrides\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-var-lib-kubelet\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-host\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-cni-multus\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351315 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 13:16:43.352372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-sys-fs\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351428 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-kubelet\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-host-var-lib-kubelet\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-netns\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-conf-dir\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwc7\" (UniqueName: \"kubernetes.io/projected/2d220c08-dc87-46d4-a868-b10a521c3b40-kube-api-access-jcwc7\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-modprobe-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e949be8-73ac-4b83-b1b6-0a1ace341746-konnectivity-ca\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-cni-binary-copy\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-system-cni-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-cnibin\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-system-cni-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-bin\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.353130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-cnibin\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01e9b533-764f-4eef-a8b1-80dc2067036a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.351981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.352097 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-os-release\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-daemon-config\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.352165 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.852142772 +0000 UTC m=+2.916491681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-os-release\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-slash\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-etc-kubernetes\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-var-lib-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fffe4b43-27dc-4eac-b530-675d9773da6f-etc-kubernetes\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-systemd\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-device-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.353916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-ovn\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-run\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c815171-5cd5-4db5-b95a-dc9395fb6000-serviceca\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01e9b533-764f-4eef-a8b1-80dc2067036a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.352605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fffe4b43-27dc-4eac-b530-675d9773da6f-multus-daemon-config\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.354772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.353069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e949be8-73ac-4b83-b1b6-0a1ace341746-konnectivity-ca\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.356440 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.356414 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e949be8-73ac-4b83-b1b6-0a1ace341746-agent-certs\") pod \"konnectivity-agent-5wbzb\" (UID: \"6e949be8-73ac-4b83-b1b6-0a1ace341746\") " pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.359760 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.359715 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kpp\" (UniqueName: \"kubernetes.io/projected/eef561c0-bb8e-48e0-8d2d-d9357819fe85-kube-api-access-z8kpp\") pod \"iptables-alerter-56xc8\" (UID: \"eef561c0-bb8e-48e0-8d2d-d9357819fe85\") " pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.360013 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.359992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsl4j\" (UniqueName: \"kubernetes.io/projected/fffe4b43-27dc-4eac-b530-675d9773da6f-kube-api-access-hsl4j\") pod \"multus-lx8xz\" (UID: \"fffe4b43-27dc-4eac-b530-675d9773da6f\") " pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.360552 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.360530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsprh\" (UniqueName: \"kubernetes.io/projected/01e9b533-764f-4eef-a8b1-80dc2067036a-kube-api-access-jsprh\") pod \"multus-additional-cni-plugins-hl96x\" (UID: \"01e9b533-764f-4eef-a8b1-80dc2067036a\") " pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.393060 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.393004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" event={"ID":"73f6bdbf997a544246c875c3b1e88545","Type":"ContainerStarted","Data":"030c0e329a289faae738204a0d91c949707e948a0a3b9c0973a9402f627e37d1"} Jul 20 13:16:43.394702 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.394672 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" event={"ID":"0a498e25e8920f5d6324995f0e2bde44","Type":"ContainerStarted","Data":"c7571eba95a2f582fab52efb6aacd3d809dd43e52edabc653acc6d19f93eb1dd"} Jul 20 13:16:43.453032 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.452992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-slash\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453032 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-var-lib-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-systemd\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-slash\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-device-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-device-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-ovn\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-run\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-var-lib-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c815171-5cd5-4db5-b95a-dc9395fb6000-serviceca\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.453223 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.453274 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-systemd\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-ovn\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-run\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-log-socket\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.453301 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.95328129 +0000 UTC m=+3.017630197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-log-socket\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d220c08-dc87-46d4-a868-b10a521c3b40-hosts-file\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-conf\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-netd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-kubernetes\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-script-lib\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mqk\" (UniqueName: \"kubernetes.io/projected/5daed80f-e228-440a-aed0-8c423a2044bf-kube-api-access-82mqk\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-kubelet\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-netd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjft\" (UniqueName: \"kubernetes.io/projected/8dec058e-f451-44de-a6a7-46f9725fd4c8-kube-api-access-ckjft\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c815171-5cd5-4db5-b95a-dc9395fb6000-host\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.453774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d220c08-dc87-46d4-a868-b10a521c3b40-hosts-file\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/9c815171-5cd5-4db5-b95a-dc9395fb6000-kube-api-access-n79mn\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-node-log\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d220c08-dc87-46d4-a868-b10a521c3b40-tmp-dir\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysconfig\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453712 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c815171-5cd5-4db5-b95a-dc9395fb6000-host\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-tuned\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-kubelet\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c815171-5cd5-4db5-b95a-dc9395fb6000-serviceca\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-systemd-units\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpm7\" (UniqueName: \"kubernetes.io/projected/11833f47-b5c7-4ad0-abf8-878cde93e3c9-kube-api-access-nqpm7\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-etc-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-registration-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453887 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-node-log\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.454671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-config\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-sys\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-lib-modules\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-tmp\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnmc\" (UniqueName: \"kubernetes.io/projected/947acb8b-57e8-4622-a44a-0b33e8c4a078-kube-api-access-jxnmc\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-socket-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovn-node-metrics-cert\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-systemd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454152 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d220c08-dc87-46d4-a868-b10a521c3b40-tmp-dir\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-env-overrides\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-var-lib-kubelet\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-host\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-systemd-units\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-sys-fs\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.455474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-etc-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.453703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-conf\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-script-lib\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454329 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-registration-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-netns\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-netns\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwc7\" (UniqueName: \"kubernetes.io/projected/2d220c08-dc87-46d4-a868-b10a521c3b40-kube-api-access-jcwc7\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-modprobe-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-socket-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-kubernetes\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-bin\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-openvswitch\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454493 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-var-lib-kubelet\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456329 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-run-systemd\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-sys\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovnkube-config\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8dec058e-f451-44de-a6a7-46f9725fd4c8-env-overrides\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455030 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5daed80f-e228-440a-aed0-8c423a2044bf-sys-fs\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-modprobe-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-host\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455104 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dec058e-f451-44de-a6a7-46f9725fd4c8-host-cni-bin\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.454263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysconfig\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455235 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-lib-modules\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.455237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-sysctl-d\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.456431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-etc-tuned\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.456993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.456902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dec058e-f451-44de-a6a7-46f9725fd4c8-ovn-node-metrics-cert\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.457566 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.457122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947acb8b-57e8-4622-a44a-0b33e8c4a078-tmp\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.464617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.464594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/9c815171-5cd5-4db5-b95a-dc9395fb6000-kube-api-access-n79mn\") pod \"node-ca-9kkq8\" (UID: \"9c815171-5cd5-4db5-b95a-dc9395fb6000\") " pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.464735 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.464680 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:43.464735 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.464701 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:43.464735 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.464713 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:43.464896 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.464789 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.964772584 +0000 UTC m=+3.029121493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:43.466277 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.466221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mqk\" (UniqueName: \"kubernetes.io/projected/5daed80f-e228-440a-aed0-8c423a2044bf-kube-api-access-82mqk\") pod \"aws-ebs-csi-driver-node-sr5f5\" (UID: \"5daed80f-e228-440a-aed0-8c423a2044bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.466969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.466910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjft\" (UniqueName: \"kubernetes.io/projected/8dec058e-f451-44de-a6a7-46f9725fd4c8-kube-api-access-ckjft\") pod \"ovnkube-node-hddgc\" (UID: \"8dec058e-f451-44de-a6a7-46f9725fd4c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.467526 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.467500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnmc\" (UniqueName: \"kubernetes.io/projected/947acb8b-57e8-4622-a44a-0b33e8c4a078-kube-api-access-jxnmc\") pod \"tuned-r5s2p\" (UID: \"947acb8b-57e8-4622-a44a-0b33e8c4a078\") " pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.467992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.467951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwc7\" (UniqueName: \"kubernetes.io/projected/2d220c08-dc87-46d4-a868-b10a521c3b40-kube-api-access-jcwc7\") pod \"node-resolver-mxjhc\" (UID: \"2d220c08-dc87-46d4-a868-b10a521c3b40\") " pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.469063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.469042 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpm7\" (UniqueName: \"kubernetes.io/projected/11833f47-b5c7-4ad0-abf8-878cde93e3c9-kube-api-access-nqpm7\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.542067 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.542032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:16:43.549828 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.549789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hl96x" Jul 20 13:16:43.556924 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.556896 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e949be8_73ac_4b83_b1b6_0a1ace341746.slice/crio-6b3b47c217de586cbc7a73d73ef76bfe780e3afef3c7f82840ff57ccf67d9d90 WatchSource:0}: Error finding container 6b3b47c217de586cbc7a73d73ef76bfe780e3afef3c7f82840ff57ccf67d9d90: Status 404 returned error can't find the container with id 6b3b47c217de586cbc7a73d73ef76bfe780e3afef3c7f82840ff57ccf67d9d90 Jul 20 13:16:43.559277 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.559220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e9b533_764f_4eef_a8b1_80dc2067036a.slice/crio-56c8859b0fcd1f3f052f94689a7c75a78cc8a428ef04d00cc661ece6ac274ec2 WatchSource:0}: Error finding container 56c8859b0fcd1f3f052f94689a7c75a78cc8a428ef04d00cc661ece6ac274ec2: Status 404 returned error can't find the container with id 56c8859b0fcd1f3f052f94689a7c75a78cc8a428ef04d00cc661ece6ac274ec2 Jul 20 13:16:43.564164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.564141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lx8xz" Jul 20 13:16:43.570141 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.570117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56xc8" Jul 20 13:16:43.575106 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.575081 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffe4b43_27dc_4eac_b530_675d9773da6f.slice/crio-b366591bf06fc1d40654f551c1cdaa02ecc7d85d95366aa0713111e88f278ef6 WatchSource:0}: Error finding container b366591bf06fc1d40654f551c1cdaa02ecc7d85d95366aa0713111e88f278ef6: Status 404 returned error can't find the container with id b366591bf06fc1d40654f551c1cdaa02ecc7d85d95366aa0713111e88f278ef6 Jul 20 13:16:43.578147 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.578126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:16:43.581040 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.581017 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef561c0_bb8e_48e0_8d2d_d9357819fe85.slice/crio-5fc257d55ffbbf8bfc654ec401904d9d4c8d862cc7a3aa952d17e3269c725508 WatchSource:0}: Error finding container 5fc257d55ffbbf8bfc654ec401904d9d4c8d862cc7a3aa952d17e3269c725508: Status 404 returned error can't find the container with id 5fc257d55ffbbf8bfc654ec401904d9d4c8d862cc7a3aa952d17e3269c725508 Jul 20 13:16:43.586025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.586003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxjhc" Jul 20 13:16:43.588363 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.588339 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dec058e_f451_44de_a6a7_46f9725fd4c8.slice/crio-4959d8e62366636b206663c1899d8870eb8c6287e162b212dbf728d48b4401f7 WatchSource:0}: Error finding container 4959d8e62366636b206663c1899d8870eb8c6287e162b212dbf728d48b4401f7: Status 404 returned error can't find the container with id 4959d8e62366636b206663c1899d8870eb8c6287e162b212dbf728d48b4401f7 Jul 20 13:16:43.594832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.594809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" Jul 20 13:16:43.597492 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.597470 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d220c08_dc87_46d4_a868_b10a521c3b40.slice/crio-fa9f422d6c6dd97094e03b304f6bb707f3ca0af576ef0a5c3478c0be7047d793 WatchSource:0}: Error finding container fa9f422d6c6dd97094e03b304f6bb707f3ca0af576ef0a5c3478c0be7047d793: Status 404 returned error can't find the container with id fa9f422d6c6dd97094e03b304f6bb707f3ca0af576ef0a5c3478c0be7047d793 Jul 20 13:16:43.603037 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.602892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9kkq8" Jul 20 13:16:43.604408 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.604372 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947acb8b_57e8_4622_a44a_0b33e8c4a078.slice/crio-ae70fd3f8559bea305f1d41bae158321829bfce6dd113e27d290b43580ad413f WatchSource:0}: Error finding container ae70fd3f8559bea305f1d41bae158321829bfce6dd113e27d290b43580ad413f: Status 404 returned error can't find the container with id ae70fd3f8559bea305f1d41bae158321829bfce6dd113e27d290b43580ad413f Jul 20 13:16:43.609146 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.609126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" Jul 20 13:16:43.614486 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.614460 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c815171_5cd5_4db5_b95a_dc9395fb6000.slice/crio-55fa32a02bae833be48f80b9ed487ae783cdf67756f62904168f00bcf372a4b6 WatchSource:0}: Error finding container 55fa32a02bae833be48f80b9ed487ae783cdf67756f62904168f00bcf372a4b6: Status 404 returned error can't find the container with id 55fa32a02bae833be48f80b9ed487ae783cdf67756f62904168f00bcf372a4b6 Jul 20 13:16:43.857199 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.857021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:43.857199 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.857152 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.857414 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.857232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:44.857213528 +0000 UTC m=+3.921562426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.958338 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:43.958307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:43.958517 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.958432 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.958517 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:43.958492 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:44.958478949 +0000 UTC m=+4.022827845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.981068 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:16:43.981042 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daed80f_e228_440a_aed0_8c423a2044bf.slice/crio-34b1ae7abb59330bd2d3803ea993467aa6c1db7ebc69f8a0525d7f965910a4b9 WatchSource:0}: Error finding container 34b1ae7abb59330bd2d3803ea993467aa6c1db7ebc69f8a0525d7f965910a4b9: Status 404 returned error can't find the container with id 34b1ae7abb59330bd2d3803ea993467aa6c1db7ebc69f8a0525d7f965910a4b9 Jul 20 13:16:44.059352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.059315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:44.059537 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.059513 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:44.059591 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.059549 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:44.059591 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.059565 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:44.059669 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.059638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:16:45.059617593 +0000 UTC m=+4.123966488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:44.307604 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.307527 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:43 +0000 UTC" deadline="2028-02-13 22:42:46.657979839 +0000 UTC" Jul 20 13:16:44.307604 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.307556 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13761h26m2.350427833s" Jul 20 13:16:44.398017 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.397978 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56xc8" event={"ID":"eef561c0-bb8e-48e0-8d2d-d9357819fe85","Type":"ContainerStarted","Data":"5fc257d55ffbbf8bfc654ec401904d9d4c8d862cc7a3aa952d17e3269c725508"} Jul 20 13:16:44.399303 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.399275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lx8xz" event={"ID":"fffe4b43-27dc-4eac-b530-675d9773da6f","Type":"ContainerStarted","Data":"b366591bf06fc1d40654f551c1cdaa02ecc7d85d95366aa0713111e88f278ef6"} Jul 20 13:16:44.400458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.400430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerStarted","Data":"56c8859b0fcd1f3f052f94689a7c75a78cc8a428ef04d00cc661ece6ac274ec2"} Jul 20 13:16:44.401706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.401678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wbzb" event={"ID":"6e949be8-73ac-4b83-b1b6-0a1ace341746","Type":"ContainerStarted","Data":"6b3b47c217de586cbc7a73d73ef76bfe780e3afef3c7f82840ff57ccf67d9d90"} Jul 20 13:16:44.403001 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.402975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" event={"ID":"5daed80f-e228-440a-aed0-8c423a2044bf","Type":"ContainerStarted","Data":"34b1ae7abb59330bd2d3803ea993467aa6c1db7ebc69f8a0525d7f965910a4b9"} Jul 20 13:16:44.404360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.404333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"4959d8e62366636b206663c1899d8870eb8c6287e162b212dbf728d48b4401f7"} Jul 20 13:16:44.405799 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.405763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9kkq8" event={"ID":"9c815171-5cd5-4db5-b95a-dc9395fb6000","Type":"ContainerStarted","Data":"55fa32a02bae833be48f80b9ed487ae783cdf67756f62904168f00bcf372a4b6"} Jul 20 13:16:44.407471 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.407447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" event={"ID":"947acb8b-57e8-4622-a44a-0b33e8c4a078","Type":"ContainerStarted","Data":"ae70fd3f8559bea305f1d41bae158321829bfce6dd113e27d290b43580ad413f"} Jul 20 13:16:44.408681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.408646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxjhc" event={"ID":"2d220c08-dc87-46d4-a868-b10a521c3b40","Type":"ContainerStarted","Data":"fa9f422d6c6dd97094e03b304f6bb707f3ca0af576ef0a5c3478c0be7047d793"} Jul 20 13:16:44.866576 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.865980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:44.866576 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.866123 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:44.866576 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.866205 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:46.866184127 +0000 UTC m=+5.930533038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:44.967641 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:44.967047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:44.967641 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.967206 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:44.967641 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:44.967280 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:46.967251101 +0000 UTC m=+6.031600009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:45.068361 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.068317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:45.068543 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.068476 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:45.068543 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.068497 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:45.068543 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.068509 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:45.068699 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.068569 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:16:47.068550539 +0000 UTC m=+6.132899457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:45.308541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.308266 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:43 +0000 UTC" deadline="2027-12-14 15:06:08.235329565 +0000 UTC" Jul 20 13:16:45.308541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.308539 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12289h49m22.926796713s" Jul 20 13:16:45.387623 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.387524 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:45.387770 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.387674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:45.387770 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.387753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:45.387912 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.387886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:45.387973 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.387951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:45.388044 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:45.388025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:45.418315 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.418216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" event={"ID":"73f6bdbf997a544246c875c3b1e88545","Type":"ContainerStarted","Data":"8a91efd312c7c70c60f6ce9ca39471c051fb845f8441d11dbb2b4764faeb47e4"} Jul 20 13:16:45.434489 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:45.434417 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-54.ec2.internal" podStartSLOduration=3.43439653 podStartE2EDuration="3.43439653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:16:45.434096325 +0000 UTC m=+4.498445241" watchObservedRunningTime="2026-07-20 13:16:45.43439653 +0000 UTC m=+4.498745449" Jul 20 13:16:46.426343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:46.426243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" event={"ID":"0a498e25e8920f5d6324995f0e2bde44","Type":"ContainerStarted","Data":"877615f14f2516abc7c5ab16f7d6b363ef06f7fc1773ff25f9b5b78694082acd"} Jul 20 13:16:46.884455 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:46.884416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:46.884605 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:46.884570 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:46.884675 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:46.884634 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:50.884617226 +0000 UTC m=+9.948966147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:46.985501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:46.985462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:46.985686 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:46.985626 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:46.985747 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:46.985690 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:50.985670223 +0000 UTC m=+10.050019129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:47.086152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.086066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:47.086343 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.086306 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:47.086343 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.086327 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:47.086343 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.086341 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:47.086494 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.086404 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:16:51.086385509 +0000 UTC m=+10.150734408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:47.387610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.386859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:47.387610 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.387004 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:47.387610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.387401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:47.387610 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.387505 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:47.388382 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.387964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:47.388382 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:47.388057 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:47.433002 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.432967 2577 generic.go:356] "Generic (PLEG): container finished" podID="0a498e25e8920f5d6324995f0e2bde44" containerID="877615f14f2516abc7c5ab16f7d6b363ef06f7fc1773ff25f9b5b78694082acd" exitCode=0 Jul 20 13:16:47.433524 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:47.433017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" event={"ID":"0a498e25e8920f5d6324995f0e2bde44","Type":"ContainerDied","Data":"877615f14f2516abc7c5ab16f7d6b363ef06f7fc1773ff25f9b5b78694082acd"} Jul 20 13:16:49.386836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:49.386801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:49.387294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:49.386838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:49.387294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:49.386806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:49.387294 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:49.386940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:49.387294 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:49.387030 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:49.387294 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:49.387119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:50.919956 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:50.919917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:50.920442 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:50.920100 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:50.920442 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:50.920206 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:58.920159515 +0000 UTC m=+17.984508416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:51.021296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:51.020723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:51.021296 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.020894 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:51.021296 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.020956 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:59.020934203 +0000 UTC m=+18.085283098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:51.121897 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:51.121748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:51.122096 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.122000 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:51.122096 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.122024 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:51.122096 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.122036 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:51.122096 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.122089 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:16:59.122072451 +0000 UTC m=+18.186421356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:51.388349 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:51.388312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:51.388532 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.388429 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:51.389031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:51.389003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:51.389145 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.389121 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:51.389236 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:51.389222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:51.389342 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:51.389300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:53.386920 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:53.386863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:53.387409 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:53.386997 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:53.387409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:53.386863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:53.387524 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:53.387479 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:53.387805 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:53.387661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:53.387805 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:53.387767 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:55.387249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:55.387217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:55.387715 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:55.387227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:55.387715 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:55.387344 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:55.387715 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:55.387227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:55.387715 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:55.387417 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:55.387715 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:55.387493 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:57.386762 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:57.386721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:57.387231 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:57.386727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:57.387231 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:57.386844 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:57.387231 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:57.386955 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:16:57.387231 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:57.386732 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:57.387231 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:57.387066 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:58.980526 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:58.980489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:58.980972 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:58.980650 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:58.980972 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:58.980721 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:14.980701094 +0000 UTC m=+34.045050021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:59.081107 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:59.081062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:59.081286 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.081195 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:59.081286 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.081261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:15.081246614 +0000 UTC m=+34.145595507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:59.181786 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:59.181741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:59.181956 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.181877 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:59.181956 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.181898 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:59.181956 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.181909 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:59.182085 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.181965 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:17:15.181949348 +0000 UTC m=+34.246298245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:59.387139 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:59.387103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:16:59.387339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:59.387117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:16:59.387339 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.387240 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:16:59.387339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:16:59.387291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:16:59.387502 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.387417 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:16:59.387559 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:16:59.387503 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:01.387595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:01.387564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:01.388111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:01.387667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:01.388111 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:01.387688 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:01.388111 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:01.387803 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:01.388111 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:01.387915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:01.388111 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:01.387992 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:03.386633 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.386608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:03.387077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.386616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:03.387077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.386775 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:03.387077 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:03.387003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:03.387356 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:03.387328 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:03.387429 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:03.387401 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:03.464946 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.464913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" event={"ID":"0a498e25e8920f5d6324995f0e2bde44","Type":"ContainerStarted","Data":"791e68e600db0fba0affb2998e37e669779907a6393514f3ae552ece710ee40e"} Jul 20 13:17:03.466380 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.466279 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" event={"ID":"5daed80f-e228-440a-aed0-8c423a2044bf","Type":"ContainerStarted","Data":"9f79eec2c1e5125220ce6ea6b3061a0122ebf047d3bd23640fbaa7a0b0bbb52b"} Jul 20 13:17:03.468027 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.468005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"732e643c10fcedc6dbd274ac32a98c94fabb09683b7d02b63674deccb6eb2fb8"} Jul 20 13:17:03.468116 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.468063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"bb054ef5f2ec1eff9ba4011b8b7d2e0c2b4251d8e2d0a03f2fe03697189a4d4c"} Jul 20 13:17:03.469439 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.469414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9kkq8" event={"ID":"9c815171-5cd5-4db5-b95a-dc9395fb6000","Type":"ContainerStarted","Data":"418ab75f02009a614242abb7ae2a0e53ddfba22c0fefbeb160952e96e440d0df"} Jul 20 13:17:03.470754 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.470728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" event={"ID":"947acb8b-57e8-4622-a44a-0b33e8c4a078","Type":"ContainerStarted","Data":"4b358ed91f3e3c1f401c51aed56dea63c116dea5c632aca83107312df0924842"} Jul 20 13:17:03.472266 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.472208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxjhc" event={"ID":"2d220c08-dc87-46d4-a868-b10a521c3b40","Type":"ContainerStarted","Data":"06a835f9040bfe78f9f90d5cf538775bca58cc153b1bacf058a1f2fda56177f7"} Jul 20 13:17:03.473435 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.473410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lx8xz" event={"ID":"fffe4b43-27dc-4eac-b530-675d9773da6f","Type":"ContainerStarted","Data":"1a4b5a94ad2af534c3a1d79685d2566c6d80b9f0c244c3c4b95eb3c5ce42d8c0"} Jul 20 13:17:03.474688 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.474665 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="86fd1e9b8e73253910680634cd276261ef97f76d595c08530484959137ab5ded" exitCode=0 Jul 20 13:17:03.474781 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.474753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"86fd1e9b8e73253910680634cd276261ef97f76d595c08530484959137ab5ded"} Jul 20 13:17:03.478695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.478676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wbzb" event={"ID":"6e949be8-73ac-4b83-b1b6-0a1ace341746","Type":"ContainerStarted","Data":"44cae15f1483b5a090126230a0e40531992cb4c7a18491f98bbad93272802ad4"} Jul 20 13:17:03.497360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.497315 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-54.ec2.internal" podStartSLOduration=21.497296277 podStartE2EDuration="21.497296277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.48196567 +0000 UTC m=+22.546314585" watchObservedRunningTime="2026-07-20 13:17:03.497296277 +0000 UTC m=+22.561645191" Jul 20 13:17:03.497584 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.497558 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9kkq8" podStartSLOduration=3.74970924 podStartE2EDuration="22.497552236s" totalImagesPullingTime="18.747842996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.497180212 +0000 UTC m=+22.561529121" watchObservedRunningTime="2026-07-20 13:17:03.497552236 +0000 UTC m=+22.561901150" Jul 20 13:17:03.546265 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.546059 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r5s2p" podStartSLOduration=3.34424692 podStartE2EDuration="22.546038332s" totalImagesPullingTime="19.201791412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.545530979 +0000 UTC m=+22.609879893" watchObservedRunningTime="2026-07-20 13:17:03.546038332 +0000 UTC m=+22.610387248" Jul 20 13:17:03.566340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.566282 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lx8xz" podStartSLOduration=3.327245233 podStartE2EDuration="22.566261356s" totalImagesPullingTime="19.239016123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.566098719 +0000 UTC m=+22.630447634" watchObservedRunningTime="2026-07-20 13:17:03.566261356 +0000 UTC m=+22.630610271" Jul 20 13:17:03.599844 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.599804 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mxjhc" podStartSLOduration=3.389690912 podStartE2EDuration="22.599789943s" totalImagesPullingTime="19.210099031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.58446424 +0000 UTC m=+22.648813155" watchObservedRunningTime="2026-07-20 13:17:03.599789943 +0000 UTC m=+22.664138858" Jul 20 13:17:03.599982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:03.599963 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5wbzb" podStartSLOduration=12.102827494 podStartE2EDuration="22.599956437s" totalImagesPullingTime="10.497128943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.599588567 +0000 UTC m=+22.663937492" watchObservedRunningTime="2026-07-20 13:17:03.599956437 +0000 UTC m=+22.664305351" Jul 20 13:17:04.483183 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483151 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483476 2577 generic.go:356] "Generic (PLEG): container finished" podID="8dec058e-f451-44de-a6a7-46f9725fd4c8" containerID="732e643c10fcedc6dbd274ac32a98c94fabb09683b7d02b63674deccb6eb2fb8" exitCode=1 Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483550 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerDied","Data":"732e643c10fcedc6dbd274ac32a98c94fabb09683b7d02b63674deccb6eb2fb8"} Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483655 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"bd8140962999a39e9b3eb57e00e0e442e80650a11f6ed38403ab204e6c79b7c8"} Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"d41ec1e6eb025cb16ab5265800f532e3c834df30ef4710084138de0b4c18f443"} Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"1fed659ad1d5cb822adc5664efe7cd85b7ea6c4576024cd65f2c6a73e090e27e"} Jul 20 13:17:04.483878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.483763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"2a2667c6d1b6934b023f56c2e1d10f2f770f60ad5889596a4819677df4f8314b"} Jul 20 13:17:04.484900 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.484880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56xc8" event={"ID":"eef561c0-bb8e-48e0-8d2d-d9357819fe85","Type":"ContainerStarted","Data":"67960f4c778cd97d9abe862891ce610e387c9e20f2aa4366971878eef340473a"} Jul 20 13:17:04.502131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.502080 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-56xc8" podStartSLOduration=4.274122433 podStartE2EDuration="23.502066646s" totalImagesPullingTime="19.227944213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:04.501795515 +0000 UTC m=+23.566144430" watchObservedRunningTime="2026-07-20 13:17:04.502066646 +0000 UTC m=+23.566415560" Jul 20 13:17:04.660721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:04.660682 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 13:17:05.311911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.311791 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T13:17:04.66070268Z","UUID":"ed05e785-a7fc-41c9-b0e8-5b6ec07b2db4","Handler":null,"Name":"","Endpoint":""} Jul 20 13:17:05.313403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.313380 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 13:17:05.313529 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.313413 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 13:17:05.387159 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.387131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:05.387333 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.387134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:05.387333 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:05.387264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:05.387412 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:05.387329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:05.387412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.387355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:05.387473 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:05.387435 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:05.488945 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:05.488910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" event={"ID":"5daed80f-e228-440a-aed0-8c423a2044bf","Type":"ContainerStarted","Data":"70e13f42c925fbddc217be4c0f57a98339f3c99e0610b04d164ab487e6e05e28"} Jul 20 13:17:06.091992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.091962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:17:06.092516 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.092501 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:17:06.495027 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.494841 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:17:06.495501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.495437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"1343da570f483d7a493dd30abaa9bcc222577eeae493a3f32d38766f4c91735c"} Jul 20 13:17:06.497734 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.497709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" event={"ID":"5daed80f-e228-440a-aed0-8c423a2044bf","Type":"ContainerStarted","Data":"69890cc75d6467d494c105bf780811782dce0fcd9a33fcf4edf31963da855bbf"} Jul 20 13:17:06.520843 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:06.520795 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr5f5" podStartSLOduration=3.737525249 podStartE2EDuration="25.520781755s" totalImagesPullingTime="21.783256506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:06.520481168 +0000 UTC m=+25.584830083" watchObservedRunningTime="2026-07-20 13:17:06.520781755 +0000 UTC m=+25.585130670" Jul 20 13:17:07.386889 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.386851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:07.387090 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.386851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:07.387090 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.386974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:07.387090 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:07.386970 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:07.387090 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:07.387049 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:07.387241 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:07.387117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:07.895232 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.895198 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:17:07.895890 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.895334 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 13:17:07.895890 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:07.895828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5wbzb" Jul 20 13:17:09.386978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.386827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:09.387500 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.386837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:09.387500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:09.387073 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:09.387500 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.386869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:09.387500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:09.387185 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:09.387500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:09.387281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:09.506011 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.505975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerStarted","Data":"2bc7365d42cf49b5ecaaec32271a47987d645631328f4da3c7b32e282d7ee6af"} Jul 20 13:17:09.509386 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.509355 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:17:09.509784 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.509753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"0a7a2fd77a220d23d2cf9e1beb2017b5140c1686b348f51183eba2e649b8f075"} Jul 20 13:17:09.509960 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.509942 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:09.510025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.510006 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:09.510071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.510054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:09.510249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.510233 2577 scope.go:117] "RemoveContainer" containerID="732e643c10fcedc6dbd274ac32a98c94fabb09683b7d02b63674deccb6eb2fb8" Jul 20 13:17:09.526985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.526959 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:09.527296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:09.527281 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:10.513595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:10.513563 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="2bc7365d42cf49b5ecaaec32271a47987d645631328f4da3c7b32e282d7ee6af" exitCode=0 Jul 20 13:17:10.514161 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:10.513653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"2bc7365d42cf49b5ecaaec32271a47987d645631328f4da3c7b32e282d7ee6af"} Jul 20 13:17:10.517501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:10.517483 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:17:10.517885 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:10.517867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" event={"ID":"8dec058e-f451-44de-a6a7-46f9725fd4c8","Type":"ContainerStarted","Data":"2f57afbe6e6e3e91b4276b87a9c79dced47dc72e660c9a1c9a709461e4405503"} Jul 20 13:17:10.569201 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:10.569142 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" podStartSLOduration=9.914359257 podStartE2EDuration="29.569126961s" totalImagesPullingTime="19.654767704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:10.567572515 +0000 UTC m=+29.631921429" watchObservedRunningTime="2026-07-20 13:17:10.569126961 +0000 UTC m=+29.633475876" Jul 20 13:17:11.388292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.388046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:11.388455 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.388104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:11.388455 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.388391 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:11.388455 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.388130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:11.388606 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.388460 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:11.388606 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.388509 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:11.438463 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.438433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtlhw"] Jul 20 13:17:11.442688 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.442644 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-52sdt"] Jul 20 13:17:11.443225 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.443203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-txwcc"] Jul 20 13:17:11.520152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.520087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:11.520152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.520104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:11.520556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:11.520087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:11.520556 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.520205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:11.520556 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.520261 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:11.520556 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:11.520324 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:12.524046 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:12.523957 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="263ea45e5ef3c3826bbfc97ee9be68a5ae096d2491b24334c51fb624c3ece87f" exitCode=0 Jul 20 13:17:12.524046 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:12.524004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"263ea45e5ef3c3826bbfc97ee9be68a5ae096d2491b24334c51fb624c3ece87f"} Jul 20 13:17:13.386581 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:13.386551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:13.386761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:13.386558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:13.386761 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:13.386659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:13.386761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:13.386667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:13.386761 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:13.386748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:13.386960 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:13.386822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:13.528823 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:13.528743 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="fe3d13dba83c15e7d92f6d8bba50eb8aacd6da420bf3da4a453cd9c24fc6c8a1" exitCode=0 Jul 20 13:17:13.528823 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:13.528801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"fe3d13dba83c15e7d92f6d8bba50eb8aacd6da420bf3da4a453cd9c24fc6c8a1"} Jul 20 13:17:14.998395 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:14.998362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:14.998848 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:14.998486 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:14.998848 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:14.998558 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret podName:6ed66974-03a7-40cc-9cea-65fc2a99cd57 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:46.998539741 +0000 UTC m=+66.062888637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret") pod "global-pull-secret-syncer-mtlhw" (UID: "6ed66974-03a7-40cc-9cea-65fc2a99cd57") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:15.099228 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:15.099135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:15.099370 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.099284 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:15.099370 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.099345 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:47.09932955 +0000 UTC m=+66.163678445 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:15.199587 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:15.199557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:15.199803 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.199726 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:17:15.199803 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.199745 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:17:15.199803 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.199755 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f28lr for pod openshift-network-diagnostics/network-check-target-txwcc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:15.199803 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.199804 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr podName:83242697-d090-4b2c-9449-e7b9f26cc06d nodeName:}" failed. No retries permitted until 2026-07-20 13:17:47.199790773 +0000 UTC m=+66.264139666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28lr" (UniqueName: "kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr") pod "network-check-target-txwcc" (UID: "83242697-d090-4b2c-9449-e7b9f26cc06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:15.387412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:15.387375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:15.387412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:15.387410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:15.387865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:15.387388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:15.387865 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.387502 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtlhw" podUID="6ed66974-03a7-40cc-9cea-65fc2a99cd57" Jul 20 13:17:15.387865 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.387596 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:17:15.387865 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:15.387679 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txwcc" podUID="83242697-d090-4b2c-9449-e7b9f26cc06d" Jul 20 13:17:16.797261 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.797182 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-54.ec2.internal" event="NodeReady" Jul 20 13:17:16.797741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.797341 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 13:17:16.835742 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.835700 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:17:16.839902 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.839879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.842643 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.842619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 13:17:16.843202 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.843165 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58ttj\"" Jul 20 13:17:16.843648 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.843629 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 13:17:16.843767 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.843670 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 13:17:16.849368 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.849336 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 13:17:16.849971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.849947 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fw6d8"] Jul 20 13:17:16.852862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.852845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:16.853396 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.853378 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2lqzk"] Jul 20 13:17:16.856227 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.856205 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 13:17:16.856323 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.856274 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:17:16.856634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.856617 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:17:16.856726 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.856717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:16.856978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.856936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 13:17:16.859610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.859590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 13:17:16.859610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.859601 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxzfq\"" Jul 20 13:17:16.859791 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.859601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 13:17:16.859791 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.859710 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 13:17:16.869957 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.869903 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fw6d8"] Jul 20 13:17:16.870671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.870649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2lqzk"] Jul 20 13:17:16.909907 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.909877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnsj\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.909907 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.909912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.909975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.909991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.910022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.910101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.910145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:16.910440 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:16.910165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011433 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011433 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qqv\" (UniqueName: \"kubernetes.io/projected/ae1b7005-8291-4dec-ae98-0f038dc793dc-kube-api-access-f6qqv\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.011556 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.011572 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1b7005-8291-4dec-ae98-0f038dc793dc-cert\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.011680 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.011636 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:17.511612717 +0000 UTC m=+36.575961616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2732cd-a004-4b76-8279-e95796aeea5f-tmp-dir\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnsj\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.011993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.011941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de2732cd-a004-4b76-8279-e95796aeea5f-config-volume\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.012352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.012000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrxt\" (UniqueName: \"kubernetes.io/projected/de2732cd-a004-4b76-8279-e95796aeea5f-kube-api-access-fqrxt\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.012352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.012081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.012787 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.012578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.012920 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.012831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.016018 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.015997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.016127 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.016018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.023858 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.023833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.024447 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.024422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnsj\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.113133 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de2732cd-a004-4b76-8279-e95796aeea5f-config-volume\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.113335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrxt\" (UniqueName: \"kubernetes.io/projected/de2732cd-a004-4b76-8279-e95796aeea5f-kube-api-access-fqrxt\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.113335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qqv\" (UniqueName: \"kubernetes.io/projected/ae1b7005-8291-4dec-ae98-0f038dc793dc-kube-api-access-f6qqv\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.113464 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1b7005-8291-4dec-ae98-0f038dc793dc-cert\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.113464 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.113464 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2732cd-a004-4b76-8279-e95796aeea5f-tmp-dir\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.113615 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.113530 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:17.113615 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.113599 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:17.613580628 +0000 UTC m=+36.677929534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:17.113797 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de2732cd-a004-4b76-8279-e95796aeea5f-config-volume\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.113869 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.113778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2732cd-a004-4b76-8279-e95796aeea5f-tmp-dir\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.116159 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.116132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1b7005-8291-4dec-ae98-0f038dc793dc-cert\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.123012 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.122988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrxt\" (UniqueName: \"kubernetes.io/projected/de2732cd-a004-4b76-8279-e95796aeea5f-kube-api-access-fqrxt\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.124500 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.124479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qqv\" (UniqueName: \"kubernetes.io/projected/ae1b7005-8291-4dec-ae98-0f038dc793dc-kube-api-access-f6qqv\") pod \"ingress-canary-2lqzk\" (UID: \"ae1b7005-8291-4dec-ae98-0f038dc793dc\") " pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.171975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.171943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2lqzk" Jul 20 13:17:17.387425 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.387346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:17.387586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.387343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:17.387586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.387478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.395186 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.395309 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.395956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.396135 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.396434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:17.396982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.396831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:17.516986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.516953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:17.517208 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.517067 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:17.517208 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.517090 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:17.517208 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.517190 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:18.517155374 +0000 UTC m=+37.581504281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:17.618419 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:17.618381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:17.618596 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.618544 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:17.618689 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:17.618617 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:18.618603074 +0000 UTC m=+37.682951970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:18.274587 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:18.274535 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2lqzk"] Jul 20 13:17:18.332906 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:17:18.332874 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1b7005_8291_4dec_ae98_0f038dc793dc.slice/crio-5d1d50e516ae8344957f7aca15c74b803f2f6cb47beb304f81bd335e44a29b11 WatchSource:0}: Error finding container 5d1d50e516ae8344957f7aca15c74b803f2f6cb47beb304f81bd335e44a29b11: Status 404 returned error can't find the container with id 5d1d50e516ae8344957f7aca15c74b803f2f6cb47beb304f81bd335e44a29b11 Jul 20 13:17:18.525533 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:18.525494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:18.525685 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:18.525629 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:18.525685 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:18.525645 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:18.525795 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:18.525708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:20.525689175 +0000 UTC m=+39.590038085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:18.544749 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:18.544718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerStarted","Data":"33521cebf9b908c81831308c710281f4fede9437a1951acc522dd3e0e2cf313b"} Jul 20 13:17:18.546556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:18.546530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2lqzk" event={"ID":"ae1b7005-8291-4dec-ae98-0f038dc793dc","Type":"ContainerStarted","Data":"5d1d50e516ae8344957f7aca15c74b803f2f6cb47beb304f81bd335e44a29b11"} Jul 20 13:17:18.626094 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:18.626063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:18.626261 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:18.626201 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:18.626320 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:18.626265 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:20.626247669 +0000 UTC m=+39.690596576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:19.552911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:19.552658 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="33521cebf9b908c81831308c710281f4fede9437a1951acc522dd3e0e2cf313b" exitCode=0 Jul 20 13:17:19.552911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:19.552747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"33521cebf9b908c81831308c710281f4fede9437a1951acc522dd3e0e2cf313b"} Jul 20 13:17:20.541305 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:20.541269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:20.541541 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:20.541443 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:20.541541 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:20.541464 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:20.541655 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:20.541552 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:24.541534487 +0000 UTC m=+43.605883384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:20.560027 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:20.559994 2577 generic.go:356] "Generic (PLEG): container finished" podID="01e9b533-764f-4eef-a8b1-80dc2067036a" containerID="53a8d853af17cdd6b82e8dd4d7afc90e59b1da90de063933c08b18138f40c862" exitCode=0 Jul 20 13:17:20.560471 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:20.560037 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerDied","Data":"53a8d853af17cdd6b82e8dd4d7afc90e59b1da90de063933c08b18138f40c862"} Jul 20 13:17:20.641986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:20.641943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:20.642150 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:20.642085 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:20.642216 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:20.642151 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:24.642131894 +0000 UTC m=+43.706480788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:21.112404 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.112367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:21.379903 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.379868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:21.511864 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.511825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:21.564083 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.564048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2lqzk" event={"ID":"ae1b7005-8291-4dec-ae98-0f038dc793dc","Type":"ContainerStarted","Data":"10c834dc642ee93113da51087456f5186c7a08012980cfb8e41d1e99292c0e09"} Jul 20 13:17:21.567315 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.567282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hl96x" event={"ID":"01e9b533-764f-4eef-a8b1-80dc2067036a","Type":"ContainerStarted","Data":"20099d54c57e28476bd02d0f02ac0bb7833cdab20491ed91ef65611406b20a8d"} Jul 20 13:17:21.594086 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.594023 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2lqzk" podStartSLOduration=2.923972376 podStartE2EDuration="5.594007808s" totalImagesPullingTime="2.670035432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:21.593574398 +0000 UTC m=+40.657923315" watchObservedRunningTime="2026-07-20 13:17:21.594007808 +0000 UTC m=+40.658356723" Jul 20 13:17:21.594784 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.594748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:21.628968 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.628920 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hl96x" podStartSLOduration=9.331712398 podStartE2EDuration="40.628905624s" totalImagesPullingTime="31.297193226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:21.628660649 +0000 UTC m=+40.693009563" watchObservedRunningTime="2026-07-20 13:17:21.628905624 +0000 UTC m=+40.693254539" Jul 20 13:17:21.798376 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.798296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:21.898535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:21.898494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:23.992433 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:23.992400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:24.060986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:24.060951 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:24.571495 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:24.571462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:24.571650 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:24.571616 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:24.571650 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:24.571635 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:24.571717 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:24.571695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:32.571679026 +0000 UTC m=+51.636027920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:24.654898 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:24.654858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:24.671938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:24.671910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:24.672060 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:24.672042 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:24.672109 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:24.672100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:32.672087627 +0000 UTC m=+51.736436524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:24.752760 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:24.752730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:25.492616 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:25.492585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:25.570812 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:25.570775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:32.629243 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:32.629205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:32.629710 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:32.629350 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:32.629710 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:32.629367 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:32.629710 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:32.629430 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:48.629413468 +0000 UTC m=+67.693762366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:32.729610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:32.729572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:32.729783 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:32.729707 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:32.729783 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:32.729775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:17:48.729756978 +0000 UTC m=+67.794105885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:34.967337 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:34.967302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:41.533365 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:41.533337 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hddgc" Jul 20 13:17:41.711148 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:41.711113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:41.798516 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:41.798432 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:43.331022 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:43.330992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:43.436878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:43.436845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:46.599190 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:46.599144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:46.688804 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:46.688774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:47.031071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.031034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:47.033910 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.033891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:47.045421 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.045397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed66974-03a7-40cc-9cea-65fc2a99cd57-original-pull-secret\") pod \"global-pull-secret-syncer-mtlhw\" (UID: \"6ed66974-03a7-40cc-9cea-65fc2a99cd57\") " pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:47.113669 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.113638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtlhw" Jul 20 13:17:47.131585 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.131557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:17:47.134536 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.134513 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:47.142803 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:47.142775 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:17:47.142914 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:47.142853 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:18:51.142836742 +0000 UTC m=+130.207185638 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : secret "metrics-daemon-secret" not found Jul 20 13:17:47.232854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.232824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:47.238349 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.238329 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:47.247200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.247153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtlhw"] Jul 20 13:17:47.249403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.249384 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:47.251316 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:17:47.251293 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed66974_03a7_40cc_9cea_65fc2a99cd57.slice/crio-43e9e6fc4626103c3923802dcffa3c107b9a51f2b4e3b2466a557e19f04df59c WatchSource:0}: Error finding container 43e9e6fc4626103c3923802dcffa3c107b9a51f2b4e3b2466a557e19f04df59c: Status 404 returned error can't find the container with id 43e9e6fc4626103c3923802dcffa3c107b9a51f2b4e3b2466a557e19f04df59c Jul 20 13:17:47.256930 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.256909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28lr\" (UniqueName: \"kubernetes.io/projected/83242697-d090-4b2c-9449-e7b9f26cc06d-kube-api-access-f28lr\") pod \"network-check-target-txwcc\" (UID: \"83242697-d090-4b2c-9449-e7b9f26cc06d\") " pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:47.409515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.409484 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:47.417723 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.417701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:47.534065 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.534025 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-txwcc"] Jul 20 13:17:47.537626 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:17:47.537602 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83242697_d090_4b2c_9449_e7b9f26cc06d.slice/crio-719ef3d6b4d8b13e97166a1882358e697ca1512f333d0fb6b6bc5e4a7fa8d338 WatchSource:0}: Error finding container 719ef3d6b4d8b13e97166a1882358e697ca1512f333d0fb6b6bc5e4a7fa8d338: Status 404 returned error can't find the container with id 719ef3d6b4d8b13e97166a1882358e697ca1512f333d0fb6b6bc5e4a7fa8d338 Jul 20 13:17:47.629724 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.629690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-txwcc" event={"ID":"83242697-d090-4b2c-9449-e7b9f26cc06d","Type":"ContainerStarted","Data":"719ef3d6b4d8b13e97166a1882358e697ca1512f333d0fb6b6bc5e4a7fa8d338"} Jul 20 13:17:47.630779 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:47.630754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtlhw" event={"ID":"6ed66974-03a7-40cc-9cea-65fc2a99cd57","Type":"ContainerStarted","Data":"43e9e6fc4626103c3923802dcffa3c107b9a51f2b4e3b2466a557e19f04df59c"} Jul 20 13:17:48.644128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:48.644090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:17:48.644577 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:48.644270 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:48.644577 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:48.644284 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:17:48.644577 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:48.644348 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:18:20.644332478 +0000 UTC m=+99.708681375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:17:48.744841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:48.744800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:17:48.745013 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:48.744941 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:48.745059 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:17:48.745031 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:18:20.745010558 +0000 UTC m=+99.809359471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:17:52.646032 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:52.645988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-txwcc" event={"ID":"83242697-d090-4b2c-9449-e7b9f26cc06d","Type":"ContainerStarted","Data":"bf96bcc7d4a8ae248699c88e589be24118d8b4429ed68c54fd9f226896c3e424"} Jul 20 13:17:52.646533 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:52.646107 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:17:52.647443 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:52.647415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtlhw" event={"ID":"6ed66974-03a7-40cc-9cea-65fc2a99cd57","Type":"ContainerStarted","Data":"6f2ce01ce6be66048954febb7c5a6947bf88f88ec63c677b84aedd7249621ccd"} Jul 20 13:17:52.671665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:52.671622 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-txwcc" podStartSLOduration=67.18804402 podStartE2EDuration="1m11.671608447s" totalImagesPullingTime="4.483564427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:52.67115779 +0000 UTC m=+71.735506705" watchObservedRunningTime="2026-07-20 13:17:52.671608447 +0000 UTC m=+71.735957363" Jul 20 13:17:56.108077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:56.108042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:17:56.261944 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:17:56.261913 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:18:20.296337 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:20.296216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:18:20.383806 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:20.383776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:18:20.657681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:20.657648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:18:20.657830 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:20.657784 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:18:20.657830 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:20.657804 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:18:20.657903 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:20.657894 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:19:24.657879465 +0000 UTC m=+163.722228359 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:18:20.758924 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:20.758897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:18:20.759036 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:20.758988 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:18:20.759036 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:20.759030 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:19:24.759018773 +0000 UTC m=+163.823367667 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:18:23.652449 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:23.652418 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-txwcc" Jul 20 13:18:23.672468 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:23.672422 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mtlhw" podStartSLOduration=97.897949834 podStartE2EDuration="1m42.672408301s" totalImagesPullingTime="4.774458467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:52.703554962 +0000 UTC m=+71.767903877" watchObservedRunningTime="2026-07-20 13:18:23.672408301 +0000 UTC m=+102.736757215" Jul 20 13:18:51.168279 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:18:51.168243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:18:51.168724 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:51.168394 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:18:51.168724 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:18:51.168464 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:20:53.168449971 +0000 UTC m=+252.232798864 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : secret "metrics-daemon-secret" not found Jul 20 13:19:19.850590 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:19.850550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" Jul 20 13:19:19.864906 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:19.864876 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fw6d8" podUID="de2732cd-a004-4b76-8279-e95796aeea5f" Jul 20 13:19:20.400051 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:20.400015 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:19:20.849211 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:19:20.849161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:19:20.849372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:19:20.849161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:19:24.683548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:19:24.683513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:19:24.684054 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:24.683635 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:19:24.684054 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:24.683647 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f796fc865-74zkf: secret "image-registry-tls" not found Jul 20 13:19:24.684054 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:24.683695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls podName:b341a888-80bf-468f-b7c8-1864e6f1ec0e nodeName:}" failed. No retries permitted until 2026-07-20 13:21:26.683680787 +0000 UTC m=+285.748029680 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls") pod "image-registry-7f796fc865-74zkf" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e") : secret "image-registry-tls" not found Jul 20 13:19:24.783777 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:19:24.783745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:19:24.783887 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:24.783872 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:19:24.783933 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:19:24.783922 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls podName:de2732cd-a004-4b76-8279-e95796aeea5f nodeName:}" failed. No retries permitted until 2026-07-20 13:21:26.783908892 +0000 UTC m=+285.848257785 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls") pod "dns-default-fw6d8" (UID: "de2732cd-a004-4b76-8279-e95796aeea5f") : secret "dns-default-metrics-tls" not found Jul 20 13:19:34.387335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:19:34.387250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:20:41.881719 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:41.881683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:20:53.263708 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:53.263668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:20:53.264064 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:53.263780 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:20:53.264064 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:53.263839 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs podName:11833f47-b5c7-4ad0-abf8-878cde93e3c9 nodeName:}" failed. No retries permitted until 2026-07-20 13:22:55.263825567 +0000 UTC m=+374.328174460 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs") pod "network-metrics-daemon-52sdt" (UID: "11833f47-b5c7-4ad0-abf8-878cde93e3c9") : secret "metrics-daemon-secret" not found Jul 20 13:20:58.354814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.354782 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt"] Jul 20 13:20:58.357656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.357639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.361017 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.360991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 20 13:20:58.361198 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.361028 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.361407 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.361072 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jmztl\"" Jul 20 13:20:58.361407 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.361072 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.362400 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.362378 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt"] Jul 20 13:20:58.363284 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.363247 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 20 13:20:58.365562 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.365544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.366372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.366356 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-bj7gj"] Jul 20 13:20:58.368953 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.368931 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 20 13:20:58.369057 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.368983 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd"] Jul 20 13:20:58.369126 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.369070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.369126 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.369079 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.369126 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.369108 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 20 13:20:58.369304 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.369188 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bnq46\"" Jul 20 13:20:58.369579 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.369556 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.372041 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.372025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 20 13:20:58.372456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.372085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt"] Jul 20 13:20:58.372456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.372110 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 20 13:20:58.372456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.372161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.372965 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.372951 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.373537 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.373518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.374845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.374801 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.375022 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.374950 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 20 13:20:58.375194 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.375162 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.375382 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.375365 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wnnmq\"" Jul 20 13:20:58.375740 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.375725 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 20 13:20:58.375818 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.375765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt"] Jul 20 13:20:58.377982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.377961 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bxg4c\"" Jul 20 13:20:58.383883 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.383866 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 20 13:20:58.385401 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.385380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd"] Jul 20 13:20:58.396429 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.396410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-bj7gj"] Jul 20 13:20:58.451255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.451231 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8"] Jul 20 13:20:58.454006 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.453992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" Jul 20 13:20:58.456355 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.456339 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5txsr\"" Jul 20 13:20:58.457984 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.457950 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp"] Jul 20 13:20:58.462734 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.462710 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-b2bdl"] Jul 20 13:20:58.463243 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.463223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" Jul 20 13:20:58.466458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.466441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.469851 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.469831 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8"] Jul 20 13:20:58.471342 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471324 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 20 13:20:58.471530 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471513 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.471627 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.471689 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471677 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.471780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471761 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2m6md\"" Jul 20 13:20:58.471780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471787 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 20 13:20:58.471915 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.471804 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-szcfs\"" Jul 20 13:20:58.473706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.473690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.477355 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.477335 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp"] Jul 20 13:20:58.481574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.481546 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 20 13:20:58.483031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.483013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-b2bdl"] Jul 20 13:20:58.500056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c16c068-807b-4d62-9829-5091e332e053-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.500156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q484\" (UniqueName: \"kubernetes.io/projected/1ab538b1-13e3-4709-90bb-05507ba05fcf-kube-api-access-8q484\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.500156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29a4494-fce7-40c5-8324-20f42b671348-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.500255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ab538b1-13e3-4709-90bb-05507ba05fcf-config\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.500255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-tmp\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ab538b1-13e3-4709-90bb-05507ba05fcf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.500345 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfph4\" (UniqueName: \"kubernetes.io/projected/e29a4494-fce7-40c5-8324-20f42b671348-kube-api-access-mfph4\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.500345 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-service-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500345 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbs8g\" (UniqueName: \"kubernetes.io/projected/66e6e7e3-929b-4928-9e5b-d3ac05572fad-kube-api-access-cbs8g\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16c068-807b-4d62-9829-5091e332e053-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.500434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7f7\" (UniqueName: \"kubernetes.io/projected/7c16c068-807b-4d62-9829-5091e332e053-kube-api-access-5p7f7\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.500434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-snapshots\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e6e7e3-929b-4928-9e5b-d3ac05572fad-serving-cert\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.500569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.500529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.593890 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.593861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:20:58.600826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.600803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q484\" (UniqueName: \"kubernetes.io/projected/1ab538b1-13e3-4709-90bb-05507ba05fcf-kube-api-access-8q484\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.600919 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.600832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29a4494-fce7-40c5-8324-20f42b671348-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.600919 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.600854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ab538b1-13e3-4709-90bb-05507ba05fcf-config\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.600989 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.600963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-trusted-ca\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.601022 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.600987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-tmp\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601022 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ab538b1-13e3-4709-90bb-05507ba05fcf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.601109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfph4\" (UniqueName: \"kubernetes.io/projected/e29a4494-fce7-40c5-8324-20f42b671348-kube-api-access-mfph4\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.601109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmxr\" (UniqueName: \"kubernetes.io/projected/66987fa4-82c7-4811-8044-124065084704-kube-api-access-nqmxr\") pod \"volume-data-source-validator-78cf4c6f6-tx2lp\" (UID: \"66987fa4-82c7-4811-8044-124065084704\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" Jul 20 13:20:58.601109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-service-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrrd\" (UniqueName: \"kubernetes.io/projected/6d3ae5bd-6c84-482f-830a-a2192710081f-kube-api-access-rlrrd\") pod \"network-check-source-7bc5b6bf85-q77w8\" (UID: \"6d3ae5bd-6c84-482f-830a-a2192710081f\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" Jul 20 13:20:58.601276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbs8g\" (UniqueName: \"kubernetes.io/projected/66e6e7e3-929b-4928-9e5b-d3ac05572fad-kube-api-access-cbs8g\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44jx\" (UniqueName: \"kubernetes.io/projected/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-kube-api-access-z44jx\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.601276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-config\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-serving-cert\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16c068-807b-4d62-9829-5091e332e053-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7f7\" (UniqueName: \"kubernetes.io/projected/7c16c068-807b-4d62-9829-5091e332e053-kube-api-access-5p7f7\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-snapshots\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-tmp\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ab538b1-13e3-4709-90bb-05507ba05fcf-config\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.601479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e6e7e3-929b-4928-9e5b-d3ac05572fad-serving-cert\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.601867 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.601867 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c16c068-807b-4d62-9829-5091e332e053-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.601867 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.601676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29a4494-fce7-40c5-8324-20f42b671348-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.601867 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:58.601849 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:20:58.602063 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:58.601940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:20:59.101903335 +0000 UTC m=+258.166252240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:20:58.602209 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.602156 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-service-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.602291 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.602186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c16c068-807b-4d62-9829-5091e332e053-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.602336 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.602292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66e6e7e3-929b-4928-9e5b-d3ac05572fad-snapshots\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.602521 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.602504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e6e7e3-929b-4928-9e5b-d3ac05572fad-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.603786 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.603765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ab538b1-13e3-4709-90bb-05507ba05fcf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.603857 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.603790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16c068-807b-4d62-9829-5091e332e053-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.604053 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.604031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66e6e7e3-929b-4928-9e5b-d3ac05572fad-serving-cert\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.610812 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.610789 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbs8g\" (UniqueName: \"kubernetes.io/projected/66e6e7e3-929b-4928-9e5b-d3ac05572fad-kube-api-access-cbs8g\") pod \"insights-operator-555d9d7885-bj7gj\" (UID: \"66e6e7e3-929b-4928-9e5b-d3ac05572fad\") " pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.611213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.611193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7f7\" (UniqueName: \"kubernetes.io/projected/7c16c068-807b-4d62-9829-5091e332e053-kube-api-access-5p7f7\") pod \"kube-storage-version-migrator-operator-7658d9b749-g6pwt\" (UID: \"7c16c068-807b-4d62-9829-5091e332e053\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.611316 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.611302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfph4\" (UniqueName: \"kubernetes.io/projected/e29a4494-fce7-40c5-8324-20f42b671348-kube-api-access-mfph4\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:58.611529 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.611514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q484\" (UniqueName: \"kubernetes.io/projected/1ab538b1-13e3-4709-90bb-05507ba05fcf-kube-api-access-8q484\") pod \"service-ca-operator-5bcbc94c78-4v8hd\" (UID: \"1ab538b1-13e3-4709-90bb-05507ba05fcf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.677838 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.677811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" Jul 20 13:20:58.683867 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.683838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" Jul 20 13:20:58.688765 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.688745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" Jul 20 13:20:58.694397 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.694371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:20:58.702379 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-trusted-ca\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.702490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmxr\" (UniqueName: \"kubernetes.io/projected/66987fa4-82c7-4811-8044-124065084704-kube-api-access-nqmxr\") pod \"volume-data-source-validator-78cf4c6f6-tx2lp\" (UID: \"66987fa4-82c7-4811-8044-124065084704\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" Jul 20 13:20:58.702490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrrd\" (UniqueName: \"kubernetes.io/projected/6d3ae5bd-6c84-482f-830a-a2192710081f-kube-api-access-rlrrd\") pod \"network-check-source-7bc5b6bf85-q77w8\" (UID: \"6d3ae5bd-6c84-482f-830a-a2192710081f\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" Jul 20 13:20:58.702490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44jx\" (UniqueName: \"kubernetes.io/projected/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-kube-api-access-z44jx\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.702667 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-config\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.702667 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.702525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-serving-cert\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.703580 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.703554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-trusted-ca\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.703730 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.703703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-config\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.704980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.704959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-serving-cert\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.718011 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.717956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmxr\" (UniqueName: \"kubernetes.io/projected/66987fa4-82c7-4811-8044-124065084704-kube-api-access-nqmxr\") pod \"volume-data-source-validator-78cf4c6f6-tx2lp\" (UID: \"66987fa4-82c7-4811-8044-124065084704\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" Jul 20 13:20:58.720128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.719938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44jx\" (UniqueName: \"kubernetes.io/projected/195a9890-4dc6-48ed-bde8-592fbfb9bf1e-kube-api-access-z44jx\") pod \"console-operator-565b7b569d-b2bdl\" (UID: \"195a9890-4dc6-48ed-bde8-592fbfb9bf1e\") " pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.720128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.720081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrrd\" (UniqueName: \"kubernetes.io/projected/6d3ae5bd-6c84-482f-830a-a2192710081f-kube-api-access-rlrrd\") pod \"network-check-source-7bc5b6bf85-q77w8\" (UID: \"6d3ae5bd-6c84-482f-830a-a2192710081f\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" Jul 20 13:20:58.763917 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.763633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" Jul 20 13:20:58.774624 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.774560 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" Jul 20 13:20:58.779337 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.779224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:20:58.831441 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.831198 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt"] Jul 20 13:20:58.838234 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:58.838199 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c16c068_807b_4d62_9829_5091e332e053.slice/crio-3d1631cf48a1be6864f985b1f54f4896e1dedb05df042f176a3e5368345fb1af WatchSource:0}: Error finding container 3d1631cf48a1be6864f985b1f54f4896e1dedb05df042f176a3e5368345fb1af: Status 404 returned error can't find the container with id 3d1631cf48a1be6864f985b1f54f4896e1dedb05df042f176a3e5368345fb1af Jul 20 13:20:58.855020 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.854944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd"] Jul 20 13:20:58.868426 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:58.868350 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab538b1_13e3_4709_90bb_05507ba05fcf.slice/crio-f52d3877fb2d9333ff80fae82a745ce72b90bdc90c82217b39655320e2a3c1fa WatchSource:0}: Error finding container f52d3877fb2d9333ff80fae82a745ce72b90bdc90c82217b39655320e2a3c1fa: Status 404 returned error can't find the container with id f52d3877fb2d9333ff80fae82a745ce72b90bdc90c82217b39655320e2a3c1fa Jul 20 13:20:58.924435 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.924408 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8"] Jul 20 13:20:58.927515 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:58.927490 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3ae5bd_6c84_482f_830a_a2192710081f.slice/crio-425e2b124cf651b834343d12b2151237ca584a607d7bb63c3a52f54dac135163 WatchSource:0}: Error finding container 425e2b124cf651b834343d12b2151237ca584a607d7bb63c3a52f54dac135163: Status 404 returned error can't find the container with id 425e2b124cf651b834343d12b2151237ca584a607d7bb63c3a52f54dac135163 Jul 20 13:20:58.941758 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.941733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp"] Jul 20 13:20:58.944457 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:58.944429 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66987fa4_82c7_4811_8044_124065084704.slice/crio-1d395d8526a2539dbbc3a5f3404b2ebd995398ec64a82cc61376027714631b0b WatchSource:0}: Error finding container 1d395d8526a2539dbbc3a5f3404b2ebd995398ec64a82cc61376027714631b0b: Status 404 returned error can't find the container with id 1d395d8526a2539dbbc3a5f3404b2ebd995398ec64a82cc61376027714631b0b Jul 20 13:20:58.957672 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:58.957649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-b2bdl"] Jul 20 13:20:58.960379 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:58.960360 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195a9890_4dc6_48ed_bde8_592fbfb9bf1e.slice/crio-c34be84e2b2e822b55f5d49d305fadd54691b57049ce2c19a5fcd23e8d0fdc5a WatchSource:0}: Error finding container c34be84e2b2e822b55f5d49d305fadd54691b57049ce2c19a5fcd23e8d0fdc5a: Status 404 returned error can't find the container with id c34be84e2b2e822b55f5d49d305fadd54691b57049ce2c19a5fcd23e8d0fdc5a Jul 20 13:20:59.056838 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.056804 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-bj7gj"] Jul 20 13:20:59.059523 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:20:59.059499 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e6e7e3_929b_4928_9e5b_d3ac05572fad.slice/crio-b875ca711aaf21cf2407c992e43fa26d6226af5fdf314f4653ccbd3fb5f4961d WatchSource:0}: Error finding container b875ca711aaf21cf2407c992e43fa26d6226af5fdf314f4653ccbd3fb5f4961d: Status 404 returned error can't find the container with id b875ca711aaf21cf2407c992e43fa26d6226af5fdf314f4653ccbd3fb5f4961d Jul 20 13:20:59.074653 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.074626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" event={"ID":"66987fa4-82c7-4811-8044-124065084704","Type":"ContainerStarted","Data":"1d395d8526a2539dbbc3a5f3404b2ebd995398ec64a82cc61376027714631b0b"} Jul 20 13:20:59.075617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.075594 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" event={"ID":"195a9890-4dc6-48ed-bde8-592fbfb9bf1e","Type":"ContainerStarted","Data":"c34be84e2b2e822b55f5d49d305fadd54691b57049ce2c19a5fcd23e8d0fdc5a"} Jul 20 13:20:59.077014 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.076991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" event={"ID":"6d3ae5bd-6c84-482f-830a-a2192710081f","Type":"ContainerStarted","Data":"6086261b6dbeac343d20921cb432796423c027f14d9f1d2d42eded040ed85e4c"} Jul 20 13:20:59.077100 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.077029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" event={"ID":"6d3ae5bd-6c84-482f-830a-a2192710081f","Type":"ContainerStarted","Data":"425e2b124cf651b834343d12b2151237ca584a607d7bb63c3a52f54dac135163"} Jul 20 13:20:59.078140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.078118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" event={"ID":"1ab538b1-13e3-4709-90bb-05507ba05fcf","Type":"ContainerStarted","Data":"f52d3877fb2d9333ff80fae82a745ce72b90bdc90c82217b39655320e2a3c1fa"} Jul 20 13:20:59.079155 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.079136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" event={"ID":"66e6e7e3-929b-4928-9e5b-d3ac05572fad","Type":"ContainerStarted","Data":"b875ca711aaf21cf2407c992e43fa26d6226af5fdf314f4653ccbd3fb5f4961d"} Jul 20 13:20:59.080178 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.080127 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" event={"ID":"7c16c068-807b-4d62-9829-5091e332e053","Type":"ContainerStarted","Data":"3d1631cf48a1be6864f985b1f54f4896e1dedb05df042f176a3e5368345fb1af"} Jul 20 13:20:59.093521 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.093481 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q77w8" podStartSLOduration=1.093469008 podStartE2EDuration="1.093469008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:20:59.093301795 +0000 UTC m=+258.157650711" watchObservedRunningTime="2026-07-20 13:20:59.093469008 +0000 UTC m=+258.157817902" Jul 20 13:20:59.106640 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:20:59.106616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:20:59.106720 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:59.106708 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:20:59.106768 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:20:59.106756 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:00.106744298 +0000 UTC m=+259.171093190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:00.115906 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:00.115871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:00.116377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:00.116030 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:00.116377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:00.116111 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:02.116091313 +0000 UTC m=+261.180440217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:02.132299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:02.132259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:02.132788 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:02.132393 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:02.132788 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:02.132448 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.132435062 +0000 UTC m=+265.196783955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:03.094812 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.094777 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" event={"ID":"66987fa4-82c7-4811-8044-124065084704","Type":"ContainerStarted","Data":"152e03fcd2c460b0f81ddad7e135d4ea5522c49be9e1cbdf30bf6d97e8e87495"} Jul 20 13:21:03.096680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.096660 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/0.log" Jul 20 13:21:03.096768 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.096690 2577 generic.go:356] "Generic (PLEG): container finished" podID="195a9890-4dc6-48ed-bde8-592fbfb9bf1e" containerID="170f082ec7b041a2ae7b3b2361a7abe992ef0cbeff3a8fcb984b11368064d0d2" exitCode=255 Jul 20 13:21:03.096768 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.096743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" event={"ID":"195a9890-4dc6-48ed-bde8-592fbfb9bf1e","Type":"ContainerDied","Data":"170f082ec7b041a2ae7b3b2361a7abe992ef0cbeff3a8fcb984b11368064d0d2"} Jul 20 13:21:03.096967 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.096955 2577 scope.go:117] "RemoveContainer" containerID="170f082ec7b041a2ae7b3b2361a7abe992ef0cbeff3a8fcb984b11368064d0d2" Jul 20 13:21:03.098308 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.098289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" event={"ID":"1ab538b1-13e3-4709-90bb-05507ba05fcf","Type":"ContainerStarted","Data":"621b136f737bb3484a5f830a4121d249c8cafd445c46d626506bbc15b528c452"} Jul 20 13:21:03.100299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.100249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" event={"ID":"66e6e7e3-929b-4928-9e5b-d3ac05572fad","Type":"ContainerStarted","Data":"8b07a672ebbe9259e8170a322d157613b587d277c27ef926093664544d6d5e68"} Jul 20 13:21:03.101845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.101823 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" event={"ID":"7c16c068-807b-4d62-9829-5091e332e053","Type":"ContainerStarted","Data":"f42f3afe29fa71f92512810a62fdfd53fa43c03c9943a01206a83f6fc317e5d2"} Jul 20 13:21:03.113976 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.113939 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-tx2lp" podStartSLOduration=1.154131948 podStartE2EDuration="5.113921801s" totalImagesPullingTime="3.959789853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:03.111612858 +0000 UTC m=+262.175961774" watchObservedRunningTime="2026-07-20 13:21:03.113921801 +0000 UTC m=+262.178270718" Jul 20 13:21:03.167215 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.167149 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" podStartSLOduration=1.09609863 podStartE2EDuration="5.16712909s" totalImagesPullingTime="4.07103046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:03.165794106 +0000 UTC m=+262.230143022" watchObservedRunningTime="2026-07-20 13:21:03.16712909 +0000 UTC m=+262.231478007" Jul 20 13:21:03.167622 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.167354 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" podStartSLOduration=1.126884169 podStartE2EDuration="5.167346223s" totalImagesPullingTime="4.040462054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:03.146001901 +0000 UTC m=+262.210350817" watchObservedRunningTime="2026-07-20 13:21:03.167346223 +0000 UTC m=+262.231695139" Jul 20 13:21:03.207071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.207044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:03.308369 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:03.308338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.107263 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107232 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:21:04.107658 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107638 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/0.log" Jul 20 13:21:04.107725 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107683 2577 generic.go:356] "Generic (PLEG): container finished" podID="195a9890-4dc6-48ed-bde8-592fbfb9bf1e" containerID="0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568" exitCode=255 Jul 20 13:21:04.107824 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" event={"ID":"195a9890-4dc6-48ed-bde8-592fbfb9bf1e","Type":"ContainerDied","Data":"0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568"} Jul 20 13:21:04.107921 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107906 2577 scope.go:117] "RemoveContainer" containerID="170f082ec7b041a2ae7b3b2361a7abe992ef0cbeff3a8fcb984b11368064d0d2" Jul 20 13:21:04.107993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.107979 2577 scope.go:117] "RemoveContainer" containerID="0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568" Jul 20 13:21:04.108580 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:04.108186 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-b2bdl_openshift-console-operator(195a9890-4dc6-48ed-bde8-592fbfb9bf1e)\"" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" podUID="195a9890-4dc6-48ed-bde8-592fbfb9bf1e" Jul 20 13:21:04.127674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.127620 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" podStartSLOduration=2.272793935 podStartE2EDuration="6.127600471s" totalImagesPullingTime="3.854806536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:03.186356645 +0000 UTC m=+262.250705558" watchObservedRunningTime="2026-07-20 13:21:04.127600471 +0000 UTC m=+263.191949387" Jul 20 13:21:04.289993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.289915 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.386286 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.386187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.487412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.487379 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.494166 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.494137 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z"] Jul 20 13:21:04.498380 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.498360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" Jul 20 13:21:04.502958 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.502936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 20 13:21:04.503075 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.502986 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 20 13:21:04.503075 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.502987 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ftbnl\"" Jul 20 13:21:04.510769 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.510750 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z"] Jul 20 13:21:04.583068 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.582967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.656462 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.656424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9q9v\" (UniqueName: \"kubernetes.io/projected/e4abd97a-e39a-44a9-950d-e7ed409720b1-kube-api-access-q9q9v\") pod \"migrator-7cc9c7ffd7-q9t5z\" (UID: \"e4abd97a-e39a-44a9-950d-e7ed409720b1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" Jul 20 13:21:04.698512 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.698480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.756782 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.756752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9q9v\" (UniqueName: \"kubernetes.io/projected/e4abd97a-e39a-44a9-950d-e7ed409720b1-kube-api-access-q9q9v\") pod \"migrator-7cc9c7ffd7-q9t5z\" (UID: \"e4abd97a-e39a-44a9-950d-e7ed409720b1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" Jul 20 13:21:04.764962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.764933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9q9v\" (UniqueName: \"kubernetes.io/projected/e4abd97a-e39a-44a9-950d-e7ed409720b1-kube-api-access-q9q9v\") pod \"migrator-7cc9c7ffd7-q9t5z\" (UID: \"e4abd97a-e39a-44a9-950d-e7ed409720b1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" Jul 20 13:21:04.798805 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.798783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:04.806790 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.806767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" Jul 20 13:21:04.945554 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.945524 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jj7mq"] Jul 20 13:21:04.949709 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.949684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:04.954515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.954494 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 13:21:04.954876 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.954847 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fctt2\"" Jul 20 13:21:04.954876 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.954862 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 13:21:04.973707 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.973663 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jj7mq"] Jul 20 13:21:04.975126 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.975101 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z"] Jul 20 13:21:04.979202 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:04.979155 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4abd97a_e39a_44a9_950d_e7ed409720b1.slice/crio-c6019a3a937683e85daa435e55cbbeafb23cfc3abeb00a16ac197563983c4801 WatchSource:0}: Error finding container c6019a3a937683e85daa435e55cbbeafb23cfc3abeb00a16ac197563983c4801: Status 404 returned error can't find the container with id c6019a3a937683e85daa435e55cbbeafb23cfc3abeb00a16ac197563983c4801 Jul 20 13:21:04.984894 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:04.984873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:05.059461 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.059433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.059584 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.059468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c86b05b-f379-4376-9275-0d09eb3be0a2-crio-socket\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.059584 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.059509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.059657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.059589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c86b05b-f379-4376-9275-0d09eb3be0a2-data-volume\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.059657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.059622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtpw\" (UniqueName: \"kubernetes.io/projected/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-api-access-zjtpw\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.106279 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.106226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:05.112835 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.112780 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:21:05.113233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.113206 2577 scope.go:117] "RemoveContainer" containerID="0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568" Jul 20 13:21:05.113388 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.113373 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-b2bdl_openshift-console-operator(195a9890-4dc6-48ed-bde8-592fbfb9bf1e)\"" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" podUID="195a9890-4dc6-48ed-bde8-592fbfb9bf1e" Jul 20 13:21:05.114125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.114103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" event={"ID":"e4abd97a-e39a-44a9-950d-e7ed409720b1","Type":"ContainerStarted","Data":"c6019a3a937683e85daa435e55cbbeafb23cfc3abeb00a16ac197563983c4801"} Jul 20 13:21:05.160679 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c86b05b-f379-4376-9275-0d09eb3be0a2-data-volume\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.160774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtpw\" (UniqueName: \"kubernetes.io/projected/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-api-access-zjtpw\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.160774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.160847 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.160828 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.160847 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c86b05b-f379-4376-9275-0d09eb3be0a2-crio-socket\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.160909 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.160878 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.660864322 +0000 UTC m=+264.725213214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.160956 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c86b05b-f379-4376-9275-0d09eb3be0a2-crio-socket\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.160956 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.160929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.161028 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.160971 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.161028 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.161014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.661000959 +0000 UTC m=+264.725349852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : configmap "kube-rbac-proxy" not found Jul 20 13:21:05.161107 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.161050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c86b05b-f379-4376-9275-0d09eb3be0a2-data-volume\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.169486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.169461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtpw\" (UniqueName: \"kubernetes.io/projected/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-api-access-zjtpw\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.664278 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.664245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.664694 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:05.664311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:05.664694 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.664407 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.664694 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.664425 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.664694 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.664468 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.664455388 +0000 UTC m=+265.728804282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : configmap "kube-rbac-proxy" not found Jul 20 13:21:05.664694 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:05.664504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.664485546 +0000 UTC m=+265.728834456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.169313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:06.169272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:06.169500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.169428 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:06.169500 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.169490 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:14.169475358 +0000 UTC m=+273.233824251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:06.632980 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:06.632946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:06.674030 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:06.674002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:06.674377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.674144 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.674377 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:06.674166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:06.674377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.674229 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.674210463 +0000 UTC m=+267.738559364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.674377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.674250 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:06.674377 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:06.674287 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.674275974 +0000 UTC m=+267.738624886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : configmap "kube-rbac-proxy" not found Jul 20 13:21:07.122749 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.122717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" event={"ID":"e4abd97a-e39a-44a9-950d-e7ed409720b1","Type":"ContainerStarted","Data":"24a891bb01b8275c182f9bdad0cf465b42d3dd07badecab8dfd0f1c2936a0535"} Jul 20 13:21:07.122933 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.122808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" event={"ID":"e4abd97a-e39a-44a9-950d-e7ed409720b1","Type":"ContainerStarted","Data":"4fd10dcf5ed40becff034007d487057b98e69cde8abbd03475b45e6fea9fb263"} Jul 20 13:21:07.143604 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.143556 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-q9t5z" podStartSLOduration=1.828217209 podStartE2EDuration="3.143541615s" totalImagesPullingTime="1.315324406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:07.142802495 +0000 UTC m=+266.207151411" watchObservedRunningTime="2026-07-20 13:21:07.143541615 +0000 UTC m=+266.207890551" Jul 20 13:21:07.293127 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.293089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:07.461417 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.461317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:07.894326 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.894298 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxjhc_2d220c08-dc87-46d4-a868-b10a521c3b40/dns-node-resolver/0.log" Jul 20 13:21:07.906523 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:07.906494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:08.494695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.494666 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9kkq8_9c815171-5cd5-4db5-b95a-dc9395fb6000/node-ca/0.log" Jul 20 13:21:08.691113 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.691079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:08.691302 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.691199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:08.691302 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:08.691217 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:08.691302 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:08.691275 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.69126028 +0000 UTC m=+271.755609176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : configmap "kube-rbac-proxy" not found Jul 20 13:21:08.691450 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:08.691313 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:08.691450 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:08.691373 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.691356952 +0000 UTC m=+271.755705851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:08.779928 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.779856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:21:08.779928 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.779923 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:21:08.780270 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:08.780258 2577 scope.go:117] "RemoveContainer" containerID="0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568" Jul 20 13:21:08.780419 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:08.780401 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-b2bdl_openshift-console-operator(195a9890-4dc6-48ed-bde8-592fbfb9bf1e)\"" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" podUID="195a9890-4dc6-48ed-bde8-592fbfb9bf1e" Jul 20 13:21:09.294803 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:09.294752 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2lqzk_ae1b7005-8291-4dec-ae98-0f038dc793dc/serve-healthcheck-canary/0.log" Jul 20 13:21:09.894163 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:09.894134 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-q9t5z_e4abd97a-e39a-44a9-950d-e7ed409720b1/migrator/0.log" Jul 20 13:21:10.094048 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.094011 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-q9t5z_e4abd97a-e39a-44a9-950d-e7ed409720b1/graceful-termination/0.log" Jul 20 13:21:10.186950 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.186874 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.287607 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.287576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.295473 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.295452 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-g6pwt_7c16c068-807b-4d62-9829-5091e332e053/kube-storage-version-migrator-operator/0.log" Jul 20 13:21:10.474222 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.474127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.575908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.575874 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.677377 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.677349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.776104 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.776024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:10.965112 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:10.965079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:11.561461 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:11.561421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:12.166751 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:12.166720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:12.589661 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:12.589582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:12.723446 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:12.723414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:12.723609 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:12.723486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:12.723609 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:12.723581 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:12.723686 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:12.723672 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.723657245 +0000 UTC m=+279.788006141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:12.723686 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:12.723588 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:12.723775 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:12.723705 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm podName:1c86b05b-f379-4376-9275-0d09eb3be0a2 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.72369486 +0000 UTC m=+279.788043757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj7mq" (UID: "1c86b05b-f379-4376-9275-0d09eb3be0a2") : configmap "kube-rbac-proxy" not found Jul 20 13:21:12.959672 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:12.959638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:13.164431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:13.164398 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:13.223972 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:13.223892 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:13.559007 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:13.558929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:14.179663 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:14.179626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:14.233986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:14.233955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:14.234126 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:14.234107 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:14.234207 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:14.234197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls podName:e29a4494-fce7-40c5-8324-20f42b671348 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:30.234164885 +0000 UTC m=+289.298513778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-55jnt" (UID: "e29a4494-fce7-40c5-8324-20f42b671348") : secret "cluster-monitoring-operator-tls" not found Jul 20 13:21:14.290286 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:14.290259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:14.870141 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:14.870108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:15.018508 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:15.018477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:15.145622 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:15.145540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:15.236489 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:15.236456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:15.413444 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:15.413370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:16.574897 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:16.574867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:16.775185 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:16.775141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:16.875818 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:16.875770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:16.980853 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:16.980814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:17.201012 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:17.200918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:17.379761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:17.379732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:17.708597 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:17.708567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:18.405293 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:18.405262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:18.774498 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:18.774420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:18.973535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:18.973502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:19.198039 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:19.197992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:19.571732 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:19.571657 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:20.387870 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.387837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:20.584596 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.584561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:20.783135 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.783060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:20.783305 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.783163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:20.783569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.783551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c86b05b-f379-4376-9275-0d09eb3be0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:20.785404 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.785383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c86b05b-f379-4376-9275-0d09eb3be0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj7mq\" (UID: \"1c86b05b-f379-4376-9275-0d09eb3be0a2\") " pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:20.798443 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.798423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:20.859634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:20.859620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jj7mq" Jul 20 13:21:21.043670 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:21.043593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jj7mq"] Jul 20 13:21:21.047329 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:21.047304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c86b05b_f379_4376_9275_0d09eb3be0a2.slice/crio-74807e52d2e29338ffab5c3891b9c58342d04f57c31e06cf4762acf5aaf77caa WatchSource:0}: Error finding container 74807e52d2e29338ffab5c3891b9c58342d04f57c31e06cf4762acf5aaf77caa: Status 404 returned error can't find the container with id 74807e52d2e29338ffab5c3891b9c58342d04f57c31e06cf4762acf5aaf77caa Jul 20 13:21:21.061469 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:21.061440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:21.169429 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:21.169397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj7mq" event={"ID":"1c86b05b-f379-4376-9275-0d09eb3be0a2","Type":"ContainerStarted","Data":"74807e52d2e29338ffab5c3891b9c58342d04f57c31e06cf4762acf5aaf77caa"} Jul 20 13:21:23.850333 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:23.850284 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" Jul 20 13:21:23.850333 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:23.850284 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fw6d8" podUID="de2732cd-a004-4b76-8279-e95796aeea5f" Jul 20 13:21:24.021395 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:24.021363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:24.115543 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:24.115504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:24.180688 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:24.180661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:24.180860 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:24.180661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:24.387806 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:24.387732 2577 scope.go:117] "RemoveContainer" containerID="0de99ea71a6edddcf8442b4532182668d6abf30aea46fbef04ffdd242e850568" Jul 20 13:21:26.188343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.188255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj7mq" event={"ID":"1c86b05b-f379-4376-9275-0d09eb3be0a2","Type":"ContainerStarted","Data":"a3526a94f2c1d2e3e6c71dc21fb32648e41126c2a29deb6b219d2db7b5282f9e"} Jul 20 13:21:26.189979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.189956 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:21:26.190096 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.190004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" event={"ID":"195a9890-4dc6-48ed-bde8-592fbfb9bf1e","Type":"ContainerStarted","Data":"39ea56cd0d129fff74493885fd985ae23c7e2fa89a351f2d6bf51baf583edc4b"} Jul 20 13:21:26.190339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.190317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:21:26.211156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.211113 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" podStartSLOduration=24.254335953000002 podStartE2EDuration="28.211099845s" totalImagesPullingTime="3.956763892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:26.210736786 +0000 UTC m=+285.275085702" watchObservedRunningTime="2026-07-20 13:21:26.211099845 +0000 UTC m=+285.275448759" Jul 20 13:21:26.739154 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.738970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:26.742455 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.742402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"image-registry-7f796fc865-74zkf\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:26.755065 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.755032 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-b2bdl" Jul 20 13:21:26.840541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.840440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:26.843221 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.843193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de2732cd-a004-4b76-8279-e95796aeea5f-metrics-tls\") pod \"dns-default-fw6d8\" (UID: \"de2732cd-a004-4b76-8279-e95796aeea5f\") " pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:26.884761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.884583 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:21:26.884761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.884723 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58ttj\"" Jul 20 13:21:26.892498 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.892138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:26.892498 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.892446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:26.945056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:26.944994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:27.046509 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.046477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:27.057628 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.057598 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fw6d8"] Jul 20 13:21:27.060722 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:27.060692 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2732cd_a004_4b76_8279_e95796aeea5f.slice/crio-d8559510696a706fa1090eb1c6144578efbfc1291391d89e34e621ddf742f43d WatchSource:0}: Error finding container d8559510696a706fa1090eb1c6144578efbfc1291391d89e34e621ddf742f43d: Status 404 returned error can't find the container with id d8559510696a706fa1090eb1c6144578efbfc1291391d89e34e621ddf742f43d Jul 20 13:21:27.078908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.078885 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:21:27.083121 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:27.083093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb341a888_80bf_468f_b7c8_1864e6f1ec0e.slice/crio-76fb660daf7e56be0003b3382d352df7753b26a11d771558bea1e22101ef8626 WatchSource:0}: Error finding container 76fb660daf7e56be0003b3382d352df7753b26a11d771558bea1e22101ef8626: Status 404 returned error can't find the container with id 76fb660daf7e56be0003b3382d352df7753b26a11d771558bea1e22101ef8626 Jul 20 13:21:27.145264 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.145230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:27.195493 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.195462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" event={"ID":"b341a888-80bf-468f-b7c8-1864e6f1ec0e","Type":"ContainerStarted","Data":"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5"} Jul 20 13:21:27.195855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.195520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" event={"ID":"b341a888-80bf-468f-b7c8-1864e6f1ec0e","Type":"ContainerStarted","Data":"76fb660daf7e56be0003b3382d352df7753b26a11d771558bea1e22101ef8626"} Jul 20 13:21:27.195855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.195568 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:27.196985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.196955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw6d8" event={"ID":"de2732cd-a004-4b76-8279-e95796aeea5f","Type":"ContainerStarted","Data":"d8559510696a706fa1090eb1c6144578efbfc1291391d89e34e621ddf742f43d"} Jul 20 13:21:27.218873 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.218685 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" podStartSLOduration=272.218668084 podStartE2EDuration="4m32.218668084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:27.217870312 +0000 UTC m=+286.282219228" watchObservedRunningTime="2026-07-20 13:21:27.218668084 +0000 UTC m=+286.283017000" Jul 20 13:21:27.270058 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.269991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:27.625262 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.625214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:27.728554 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:27.728516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:28.125975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:28.125918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:28.229403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:28.229373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:28.327422 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:28.327391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:28.431353 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:28.431276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:28.835372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:28.835298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:29.113235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.113127 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:21:29.125670 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.125641 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-89sfj"] Jul 20 13:21:29.136861 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.136836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:29.140134 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.140056 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 13:21:29.140134 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.140090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 13:21:29.142397 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.142378 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-62cch\"" Jul 20 13:21:29.149483 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.148637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-89sfj"] Jul 20 13:21:29.165275 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.165241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:29.215522 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.215491 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c9d4cb47f-sm529"] Jul 20 13:21:29.220236 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.220214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.241931 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.241906 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9d4cb47f-sm529"] Jul 20 13:21:29.262501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-ca-trust-extracted\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-certificates\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-bound-sa-token\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzm9\" (UniqueName: \"kubernetes.io/projected/54a75e55-a4bc-4576-b5a7-34de68eb6ec8-kube-api-access-hhzm9\") pod \"downloads-5bc67dcd46-89sfj\" (UID: \"54a75e55-a4bc-4576-b5a7-34de68eb6ec8\") " pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:29.262656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-installation-pull-secrets\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-tls\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdww\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-kube-api-access-5bdww\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-trusted-ca\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.262908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.262778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-image-registry-private-configuration\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.273751 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.273675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:29.364104 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-trusted-ca\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364104 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-image-registry-private-configuration\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-ca-trust-extracted\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-certificates\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-bound-sa-token\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzm9\" (UniqueName: \"kubernetes.io/projected/54a75e55-a4bc-4576-b5a7-34de68eb6ec8-kube-api-access-hhzm9\") pod \"downloads-5bc67dcd46-89sfj\" (UID: \"54a75e55-a4bc-4576-b5a7-34de68eb6ec8\") " pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-installation-pull-secrets\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-tls\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdww\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-kube-api-access-5bdww\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.364713 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.364526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-ca-trust-extracted\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.365116 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.365069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-trusted-ca\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.365263 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.365240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-certificates\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.367238 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.367215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-registry-tls\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.367467 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.367311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-installation-pull-secrets\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.367680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.367656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-image-registry-private-configuration\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.373835 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.373810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-bound-sa-token\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.375299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.375275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdww\" (UniqueName: \"kubernetes.io/projected/d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4-kube-api-access-5bdww\") pod \"image-registry-6c9d4cb47f-sm529\" (UID: \"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4\") " pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.378282 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.378259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzm9\" (UniqueName: \"kubernetes.io/projected/54a75e55-a4bc-4576-b5a7-34de68eb6ec8-kube-api-access-hhzm9\") pod \"downloads-5bc67dcd46-89sfj\" (UID: \"54a75e55-a4bc-4576-b5a7-34de68eb6ec8\") " pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:29.441412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.441339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:29.450224 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.450198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:29.530478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.530444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:29.625262 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:29.625185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-89sfj"] Jul 20 13:21:29.986428 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:29.986361 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a75e55_a4bc_4576_b5a7_34de68eb6ec8.slice/crio-9b201a996d573ce50c080127df2a6bf7a8cf52f217e244a527982028e6710013 WatchSource:0}: Error finding container 9b201a996d573ce50c080127df2a6bf7a8cf52f217e244a527982028e6710013: Status 404 returned error can't find the container with id 9b201a996d573ce50c080127df2a6bf7a8cf52f217e244a527982028e6710013 Jul 20 13:21:30.065419 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.065387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:30.127957 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.127930 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9d4cb47f-sm529"] Jul 20 13:21:30.130985 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:30.130946 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e92ed2_8df0_44b3_ab24_b9eee7e0b7c4.slice/crio-9e10e455f19d6507561068e6a76fe7500bcc5fe1bd12222d9536d9fcec0f55f0 WatchSource:0}: Error finding container 9e10e455f19d6507561068e6a76fe7500bcc5fe1bd12222d9536d9fcec0f55f0: Status 404 returned error can't find the container with id 9e10e455f19d6507561068e6a76fe7500bcc5fe1bd12222d9536d9fcec0f55f0 Jul 20 13:21:30.170669 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.170597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:30.210070 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.209998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj7mq" event={"ID":"1c86b05b-f379-4376-9275-0d09eb3be0a2","Type":"ContainerStarted","Data":"3753fe1b8f9477924b5182eaabd5b6e204eca45b54cf7b333c66743146f801d2"} Jul 20 13:21:30.211371 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.211344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw6d8" event={"ID":"de2732cd-a004-4b76-8279-e95796aeea5f","Type":"ContainerStarted","Data":"c1e5846bbfa516f7799e7847349f5a1aabf5f817963182cfc155efe5113d2b5e"} Jul 20 13:21:30.213019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.212995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" event={"ID":"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4","Type":"ContainerStarted","Data":"4b01750fc239ce0a0f8ca3ec74c5082287d08f1a7a00f68e6e6e15d945478196"} Jul 20 13:21:30.213136 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.213037 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" event={"ID":"d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4","Type":"ContainerStarted","Data":"9e10e455f19d6507561068e6a76fe7500bcc5fe1bd12222d9536d9fcec0f55f0"} Jul 20 13:21:30.213136 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.213063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:30.214162 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.214143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-89sfj" event={"ID":"54a75e55-a4bc-4576-b5a7-34de68eb6ec8","Type":"ContainerStarted","Data":"9b201a996d573ce50c080127df2a6bf7a8cf52f217e244a527982028e6710013"} Jul 20 13:21:30.236402 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.236356 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" podStartSLOduration=1.236339897 podStartE2EDuration="1.236339897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:30.235098339 +0000 UTC m=+289.299447256" watchObservedRunningTime="2026-07-20 13:21:30.236339897 +0000 UTC m=+289.300688857" Jul 20 13:21:30.273848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.273598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:30.275855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.275830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29a4494-fce7-40c5-8324-20f42b671348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-55jnt\" (UID: \"e29a4494-fce7-40c5-8324-20f42b671348\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:30.467931 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.467895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" Jul 20 13:21:30.618661 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:30.618633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt"] Jul 20 13:21:30.622134 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:30.622105 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29a4494_fce7_40c5_8324_20f42b671348.slice/crio-98b5b95741b4058590d567a2387804d9ee76dfe3d317283451366553e4e764f7 WatchSource:0}: Error finding container 98b5b95741b4058590d567a2387804d9ee76dfe3d317283451366553e4e764f7: Status 404 returned error can't find the container with id 98b5b95741b4058590d567a2387804d9ee76dfe3d317283451366553e4e764f7 Jul 20 13:21:31.221645 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:31.221604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw6d8" event={"ID":"de2732cd-a004-4b76-8279-e95796aeea5f","Type":"ContainerStarted","Data":"c171b4b7a071c6683dba5ff7b9322ee7500f445c62c67fb7d7f9ccc007046342"} Jul 20 13:21:31.221816 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:31.221686 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:31.223233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:31.223152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" event={"ID":"e29a4494-fce7-40c5-8324-20f42b671348","Type":"ContainerStarted","Data":"98b5b95741b4058590d567a2387804d9ee76dfe3d317283451366553e4e764f7"} Jul 20 13:21:31.242212 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:31.242143 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fw6d8" podStartSLOduration=252.267229311 podStartE2EDuration="4m15.242123426s" totalImagesPullingTime="2.974894115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:31.241331321 +0000 UTC m=+290.305680234" watchObservedRunningTime="2026-07-20 13:21:31.242123426 +0000 UTC m=+290.306472342" Jul 20 13:21:35.066549 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.066475 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:35.168340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.168305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:35.243594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.243555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" event={"ID":"e29a4494-fce7-40c5-8324-20f42b671348","Type":"ContainerStarted","Data":"37e5ef1b88a4c1e02ebfc6ac8a2cf80b3106482429f8eba2fa0b268a334aa5eb"} Jul 20 13:21:35.246206 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.246026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj7mq" event={"ID":"1c86b05b-f379-4376-9275-0d09eb3be0a2","Type":"ContainerStarted","Data":"38389ee48cc62cb5141855424974d09882e8deeff569356690b756d37033320e"} Jul 20 13:21:35.252209 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.252163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:35.266717 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.266671 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-55jnt" podStartSLOduration=33.237047943 podStartE2EDuration="37.266653383s" totalImagesPullingTime="4.02960544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:35.266224223 +0000 UTC m=+294.330573141" watchObservedRunningTime="2026-07-20 13:21:35.266653383 +0000 UTC m=+294.331002299" Jul 20 13:21:35.288958 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.288907 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jj7mq" podStartSLOduration=17.833802411 podStartE2EDuration="31.288889385s" totalImagesPullingTime="13.455086974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:35.286858398 +0000 UTC m=+294.351207315" watchObservedRunningTime="2026-07-20 13:21:35.288889385 +0000 UTC m=+294.353238302" Jul 20 13:21:35.770192 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:35.770131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:37.388124 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:37.388049 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-52sdt" podUID="11833f47-b5c7-4ad0-abf8-878cde93e3c9" Jul 20 13:21:38.735331 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:38.735294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:38.836218 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:38.836163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:39.300596 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:39.300559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:39.396842 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:39.396803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:39.860889 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:39.860849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:41.231514 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:41.231485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fw6d8" Jul 20 13:21:41.958424 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:41.958382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:42.687762 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:42.687725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:42.818743 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:42.818709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:43.468559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.468528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:43.711342 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.711263 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ntqfq"] Jul 20 13:21:43.717375 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.717349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.721540 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.721473 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzg4g\"" Jul 20 13:21:43.721671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.721639 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 13:21:43.722293 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.722271 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 13:21:43.722392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.722315 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 13:21:43.722392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.722328 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 13:21:43.787640 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.787612 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-j98nd"] Jul 20 13:21:43.790487 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxsz\" (UniqueName: \"kubernetes.io/projected/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-kube-api-access-4lxsz\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-sys\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-wtmp\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-textfile\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-root\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.790814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.790783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-metrics-client-ca\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.791516 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.791485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.794325 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.794301 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 20 13:21:43.794434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.794350 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 20 13:21:43.794434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.794361 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wwnhq\"" Jul 20 13:21:43.794991 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.794969 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 20 13:21:43.811818 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.811795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-j98nd"] Jul 20 13:21:43.892071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-wtmp\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-textfile\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-root\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-metrics-client-ca\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-root\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxsz\" (UniqueName: \"kubernetes.io/projected/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-kube-api-access-4lxsz\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-wtmp\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:43.892269 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 20 13:21:43.892292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:43.892332 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls podName:9e35aaf1-cf04-4796-b4f5-b098d2d9f66c nodeName:}" failed. No retries permitted until 2026-07-20 13:21:44.392316775 +0000 UTC m=+303.456665673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls") pod "node-exporter-ntqfq" (UID: "9e35aaf1-cf04-4796-b4f5-b098d2d9f66c") : secret "node-exporter-tls" not found Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-sys\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/89fad513-93c8-48b8-9cb7-df45b04bb0e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-sys\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-textfile\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brr4x\" (UniqueName: \"kubernetes.io/projected/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-api-access-brr4x\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.892793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.893245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.892967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-metrics-client-ca\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.893245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.893050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.894884 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.894860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.914811 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.914789 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxsz\" (UniqueName: \"kubernetes.io/projected/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-kube-api-access-4lxsz\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:43.993960 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.993816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.993960 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.993887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.993960 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.993922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.993960 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.993954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/89fad513-93c8-48b8-9cb7-df45b04bb0e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.994283 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.993983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brr4x\" (UniqueName: \"kubernetes.io/projected/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-api-access-brr4x\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.994283 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.994017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.994606 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.994550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/89fad513-93c8-48b8-9cb7-df45b04bb0e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.994743 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.994684 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.994864 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.994843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89fad513-93c8-48b8-9cb7-df45b04bb0e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.996978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.996943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:43.997102 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:43.997026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:44.003209 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:44.003160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brr4x\" (UniqueName: \"kubernetes.io/projected/89fad513-93c8-48b8-9cb7-df45b04bb0e3-kube-api-access-brr4x\") pod \"kube-state-metrics-56c5594674-j98nd\" (UID: \"89fad513-93c8-48b8-9cb7-df45b04bb0e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:44.102007 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:44.101972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" Jul 20 13:21:44.398082 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:44.398051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:44.400666 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:44.400635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e35aaf1-cf04-4796-b4f5-b098d2d9f66c-node-exporter-tls\") pod \"node-exporter-ntqfq\" (UID: \"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c\") " pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:44.628218 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:44.628163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ntqfq" Jul 20 13:21:45.646729 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:45.646683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:45.815962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:45.815923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:45.914845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:45.914765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:46.011500 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:46.011464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:46.701480 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:46.701446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:46.810353 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:46.810312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:46.892131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:46.892088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:47.068289 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.068215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:47.091720 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:47.091689 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e35aaf1_cf04_4796_b4f5_b098d2d9f66c.slice/crio-536a764d3aeeb2a3a928cf4a416c0ce6e02eaf407e0bf4faf4e1158658f73872 WatchSource:0}: Error finding container 536a764d3aeeb2a3a928cf4a416c0ce6e02eaf407e0bf4faf4e1158658f73872: Status 404 returned error can't find the container with id 536a764d3aeeb2a3a928cf4a416c0ce6e02eaf407e0bf4faf4e1158658f73872 Jul 20 13:21:47.097078 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.097060 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:21:47.108157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.108135 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:21:47.108690 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.108671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:21:47.113975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.113953 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:21:47.114802 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.114766 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:21:47.120484 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.120469 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 13:21:47.222079 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.222057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-j98nd"] Jul 20 13:21:47.224567 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:47.224542 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fad513_93c8_48b8_9cb7_df45b04bb0e3.slice/crio-c7f7396c3602027b4f6b03ace17c161f7254d18f2986326d1c01fd598cac6a1b WatchSource:0}: Error finding container c7f7396c3602027b4f6b03ace17c161f7254d18f2986326d1c01fd598cac6a1b: Status 404 returned error can't find the container with id c7f7396c3602027b4f6b03ace17c161f7254d18f2986326d1c01fd598cac6a1b Jul 20 13:21:47.294230 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.294203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntqfq" event={"ID":"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c","Type":"ContainerStarted","Data":"536a764d3aeeb2a3a928cf4a416c0ce6e02eaf407e0bf4faf4e1158658f73872"} Jul 20 13:21:47.295452 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.295419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" event={"ID":"89fad513-93c8-48b8-9cb7-df45b04bb0e3","Type":"ContainerStarted","Data":"c7f7396c3602027b4f6b03ace17c161f7254d18f2986326d1c01fd598cac6a1b"} Jul 20 13:21:47.296719 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.296699 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-89sfj" event={"ID":"54a75e55-a4bc-4576-b5a7-34de68eb6ec8","Type":"ContainerStarted","Data":"f0877c8be18b7d3339cdd354370ae0d3885bf22b8476678890d3f6f3ea4fafc7"} Jul 20 13:21:47.296938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.296919 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:47.298399 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.298376 2577 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-89sfj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused" start-of-body= Jul 20 13:21:47.298478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.298416 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-89sfj" podUID="54a75e55-a4bc-4576-b5a7-34de68eb6ec8" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused" Jul 20 13:21:47.330483 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:47.330380 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-89sfj" podStartSLOduration=1.091295719 podStartE2EDuration="18.330363628s" totalImagesPullingTime="17.239067909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:47.329915332 +0000 UTC m=+306.394264246" watchObservedRunningTime="2026-07-20 13:21:47.330363628 +0000 UTC m=+306.394712543" Jul 20 13:21:48.121130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.121087 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7dfdcb75f-45zq8"] Jul 20 13:21:48.125958 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.125932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.128938 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.129229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.129331 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.129461 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hvcln\"" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.129539 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 20 13:21:48.129763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.129633 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dcvjm6n6q7u2n\"" Jul 20 13:21:48.139805 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.139782 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dfdcb75f-45zq8"] Jul 20 13:21:48.160288 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.160244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:48.233631 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.233583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:48.235938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.235862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvk7\" (UniqueName: \"kubernetes.io/projected/060566f1-4344-483e-b040-edddc62d9a04-kube-api-access-kzvk7\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.235938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.235913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-client-certs\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.235938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.235963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/060566f1-4344-483e-b040-edddc62d9a04-audit-log\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.236378 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.236019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-tls\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.236378 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.236064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-metrics-server-audit-profiles\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.236378 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.236240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-client-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.236378 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.236283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.312485 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.312452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-89sfj" Jul 20 13:21:48.337411 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.337339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-metrics-server-audit-profiles\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.337634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.337528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-client-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.337634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.337568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.338775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.337862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvk7\" (UniqueName: \"kubernetes.io/projected/060566f1-4344-483e-b040-edddc62d9a04-kube-api-access-kzvk7\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.338775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.338483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-client-certs\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.338775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.338553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/060566f1-4344-483e-b040-edddc62d9a04-audit-log\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.338775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.338589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-tls\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.338775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.338680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-metrics-server-audit-profiles\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.339405 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.339295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/060566f1-4344-483e-b040-edddc62d9a04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.339916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.339893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/060566f1-4344-483e-b040-edddc62d9a04-audit-log\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.342458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.342426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-client-ca-bundle\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.343300 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.343254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-tls\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.344314 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.344290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/060566f1-4344-483e-b040-edddc62d9a04-secret-metrics-server-client-certs\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.359501 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.359452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvk7\" (UniqueName: \"kubernetes.io/projected/060566f1-4344-483e-b040-edddc62d9a04-kube-api-access-kzvk7\") pod \"metrics-server-7dfdcb75f-45zq8\" (UID: \"060566f1-4344-483e-b040-edddc62d9a04\") " pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.442302 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.442158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:21:48.503365 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.503323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:48.503535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.503462 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx"] Jul 20 13:21:48.522989 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.522959 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx"] Jul 20 13:21:48.523125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.523095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.526038 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.526012 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 20 13:21:48.526517 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.526493 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wvchd\"" Jul 20 13:21:48.585333 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.585298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:48.642901 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.642821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40583dc8-d7f3-4536-8782-34f8e48f2bba-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.643078 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.642960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhsr\" (UniqueName: \"kubernetes.io/projected/40583dc8-d7f3-4536-8782-34f8e48f2bba-kube-api-access-cxhsr\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.672955 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.672552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:48.691213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.691129 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dfdcb75f-45zq8"] Jul 20 13:21:48.744394 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.744313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40583dc8-d7f3-4536-8782-34f8e48f2bba-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.744558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.744434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhsr\" (UniqueName: \"kubernetes.io/projected/40583dc8-d7f3-4536-8782-34f8e48f2bba-kube-api-access-cxhsr\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.747740 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.747708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40583dc8-d7f3-4536-8782-34f8e48f2bba-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.757009 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.756960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhsr\" (UniqueName: \"kubernetes.io/projected/40583dc8-d7f3-4536-8782-34f8e48f2bba-kube-api-access-cxhsr\") pod \"monitoring-plugin-7b7b7bc8bf-5wvcx\" (UID: \"40583dc8-d7f3-4536-8782-34f8e48f2bba\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:48.836322 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:48.836060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:49.213725 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:49.213692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:49.262104 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:49.262068 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060566f1_4344_483e_b040_edddc62d9a04.slice/crio-e370d62f9c7c5bc78e53a5219ce90c002d2afbd3c0a0e0c5d67fa3951d52fe6f WatchSource:0}: Error finding container e370d62f9c7c5bc78e53a5219ce90c002d2afbd3c0a0e0c5d67fa3951d52fe6f: Status 404 returned error can't find the container with id e370d62f9c7c5bc78e53a5219ce90c002d2afbd3c0a0e0c5d67fa3951d52fe6f Jul 20 13:21:49.307250 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:49.307204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" event={"ID":"060566f1-4344-483e-b040-edddc62d9a04","Type":"ContainerStarted","Data":"e370d62f9c7c5bc78e53a5219ce90c002d2afbd3c0a0e0c5d67fa3951d52fe6f"} Jul 20 13:21:49.618153 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:49.618123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx"] Jul 20 13:21:49.621028 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:21:49.620998 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40583dc8_d7f3_4536_8782_34f8e48f2bba.slice/crio-d2062b02e261dc378d3dc6b24696a3c58fad493f088a14b2bb23e1e296f2ad98 WatchSource:0}: Error finding container d2062b02e261dc378d3dc6b24696a3c58fad493f088a14b2bb23e1e296f2ad98: Status 404 returned error can't find the container with id d2062b02e261dc378d3dc6b24696a3c58fad493f088a14b2bb23e1e296f2ad98 Jul 20 13:21:50.050743 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.050668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:50.146537 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.146500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:50.314079 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.313997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" event={"ID":"89fad513-93c8-48b8-9cb7-df45b04bb0e3","Type":"ContainerStarted","Data":"ab52dca0bea300b11c709c6aad982e14fa5398e793312b422115b168b2afc5f7"} Jul 20 13:21:50.314079 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.314066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" event={"ID":"89fad513-93c8-48b8-9cb7-df45b04bb0e3","Type":"ContainerStarted","Data":"cfc10bbafb9ba7b98c3583a3897f109776b0fe86657813d0ce2a6a913cbbb7a1"} Jul 20 13:21:50.314625 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.314097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" event={"ID":"89fad513-93c8-48b8-9cb7-df45b04bb0e3","Type":"ContainerStarted","Data":"1cf6ce58e107cf5cda72d1aee5b98857c88b2b0ef8c526e614197d7eece98e44"} Jul 20 13:21:50.315386 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.315356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" event={"ID":"40583dc8-d7f3-4536-8782-34f8e48f2bba","Type":"ContainerStarted","Data":"d2062b02e261dc378d3dc6b24696a3c58fad493f088a14b2bb23e1e296f2ad98"} Jul 20 13:21:50.317142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.317115 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e35aaf1-cf04-4796-b4f5-b098d2d9f66c" containerID="56fbe11fd23af2b6e0d98f57ed84123037545a5ee40a22ddd4ad65e3641893cf" exitCode=0 Jul 20 13:21:50.317255 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.317200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntqfq" event={"ID":"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c","Type":"ContainerDied","Data":"56fbe11fd23af2b6e0d98f57ed84123037545a5ee40a22ddd4ad65e3641893cf"} Jul 20 13:21:50.338545 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:50.338492 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-j98nd" podStartSLOduration=5.120043809 podStartE2EDuration="7.338448705s" totalImagesPullingTime="2.218404896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:50.336079704 +0000 UTC m=+309.400428620" watchObservedRunningTime="2026-07-20 13:21:50.338448705 +0000 UTC m=+309.402797622" Jul 20 13:21:51.229919 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.229888 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c9d4cb47f-sm529" Jul 20 13:21:51.326903 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.326629 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntqfq" event={"ID":"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c","Type":"ContainerStarted","Data":"c1f9279b0969337dcfb0ab05ff35d5821828c90915d850a4ab5db0cc335f408e"} Jul 20 13:21:51.326903 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.326687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ntqfq" event={"ID":"9e35aaf1-cf04-4796-b4f5-b098d2d9f66c","Type":"ContainerStarted","Data":"0397ce715c985f2ae90acce8785dd52830fb21292ffcfb472c563bce035890ed"} Jul 20 13:21:51.429697 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.429637 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ntqfq" podStartSLOduration=5.927649879 podStartE2EDuration="8.42961605s" totalImagesPullingTime="2.348037212s" totalInitContainerRuntime="153.928959ms" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:51.427650418 +0000 UTC m=+310.491999338" watchObservedRunningTime="2026-07-20 13:21:51.42961605 +0000 UTC m=+310.493964966" Jul 20 13:21:51.589004 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.588889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:51.693901 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.693863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:51.797155 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:51.797117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:52.387227 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:52.387193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:21:53.336897 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.336852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" event={"ID":"40583dc8-d7f3-4536-8782-34f8e48f2bba","Type":"ContainerStarted","Data":"259c70430e3b41b4e8b6682cdc7b24f70b20cffd098098f0acd8501b26f8cd4c"} Jul 20 13:21:53.337095 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.336960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:53.338789 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.338760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" event={"ID":"060566f1-4344-483e-b040-edddc62d9a04","Type":"ContainerStarted","Data":"5c193d666570e165bd53fc854c26d74bed9f7c837501ead35b6f83efb0f61e05"} Jul 20 13:21:53.343087 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.343063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" Jul 20 13:21:53.356751 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.356695 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-5wvcx" podStartSLOduration=2.696593902 podStartE2EDuration="5.356676923s" totalImagesPullingTime="2.660083021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:53.355031948 +0000 UTC m=+312.419380863" watchObservedRunningTime="2026-07-20 13:21:53.356676923 +0000 UTC m=+312.421025840" Jul 20 13:21:53.416063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:53.415997 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" podStartSLOduration=2.213526942 podStartE2EDuration="5.415978211s" totalImagesPullingTime="3.202451269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:53.414746434 +0000 UTC m=+312.479095348" watchObservedRunningTime="2026-07-20 13:21:53.415978211 +0000 UTC m=+312.480327126" Jul 20 13:21:54.067442 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.067406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:54.143969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.143931 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:54.234549 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.234503 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" containerName="registry" containerID="cri-o://812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5" gracePeriod=30 Jul 20 13:21:54.545590 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.545553 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:54.705968 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.705872 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.705968 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.705940 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rnsj\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.705968 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.705973 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706018 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706049 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706072 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706454 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca\") pod \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\" (UID: \"b341a888-80bf-468f-b7c8-1864e6f1ec0e\") " Jul 20 13:21:54.706962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706902 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:21:54.706962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.706913 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:21:54.708795 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.708526 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:21:54.708795 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.708649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:21:54.709140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.709097 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:21:54.709140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.709128 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj" (OuterVolumeSpecName: "kube-api-access-8rnsj") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "kube-api-access-8rnsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:21:54.709297 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.709213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:21:54.719523 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.719493 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b341a888-80bf-468f-b7c8-1864e6f1ec0e" (UID: "b341a888-80bf-468f-b7c8-1864e6f1ec0e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:21:54.808046 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808005 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-certificates\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808046 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808043 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b341a888-80bf-468f-b7c8-1864e6f1ec0e-trusted-ca\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808061 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-bound-sa-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808075 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rnsj\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-kube-api-access-8rnsj\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808088 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b341a888-80bf-468f-b7c8-1864e6f1ec0e-registry-tls\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808103 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b341a888-80bf-468f-b7c8-1864e6f1ec0e-ca-trust-extracted\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808117 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-installation-pull-secrets\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:54.808312 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:54.808133 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b341a888-80bf-468f-b7c8-1864e6f1ec0e-image-registry-private-configuration\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:21:55.293730 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.293666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:55.349986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.349954 2577 generic.go:356] "Generic (PLEG): container finished" podID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" containerID="812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5" exitCode=0 Jul 20 13:21:55.350157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.350003 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" Jul 20 13:21:55.350157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.350039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" event={"ID":"b341a888-80bf-468f-b7c8-1864e6f1ec0e","Type":"ContainerDied","Data":"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5"} Jul 20 13:21:55.350157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.350092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f796fc865-74zkf" event={"ID":"b341a888-80bf-468f-b7c8-1864e6f1ec0e","Type":"ContainerDied","Data":"76fb660daf7e56be0003b3382d352df7753b26a11d771558bea1e22101ef8626"} Jul 20 13:21:55.350157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.350118 2577 scope.go:117] "RemoveContainer" containerID="812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5" Jul 20 13:21:55.360444 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.360405 2577 scope.go:117] "RemoveContainer" containerID="812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5" Jul 20 13:21:55.360729 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:21:55.360704 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5\": container with ID starting with 812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5 not found: ID does not exist" containerID="812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5" Jul 20 13:21:55.360815 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.360739 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5"} err="failed to get container status \"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5\": rpc error: code = NotFound desc = could not find container \"812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5\": container with ID starting with 812a29d0d144021ff5678faad3365cf7c09768d8d344f9ccca90226035a355f5 not found: ID does not exist" Jul 20 13:21:55.374775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.374749 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:21:55.378584 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.378558 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f796fc865-74zkf"] Jul 20 13:21:55.392972 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:55.392940 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" path="/var/lib/kubelet/pods/b341a888-80bf-468f-b7c8-1864e6f1ec0e/volumes" Jul 20 13:21:59.072365 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:59.072328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:59.152473 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:59.152438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:21:59.984128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:21:59.984088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:00.090816 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:00.090772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:00.206363 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:00.206330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:00.799092 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:00.799060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:00.896300 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:00.896270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:01.172785 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:01.172748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:01.246888 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:01.246857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:08.442629 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:08.442594 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:22:08.443097 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:08.442652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:22:09.014113 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:09.014068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:09.111185 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:09.111135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:09.400783 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:09.400749 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c16c068-807b-4d62-9829-5091e332e053" containerID="f42f3afe29fa71f92512810a62fdfd53fa43c03c9943a01206a83f6fc317e5d2" exitCode=0 Jul 20 13:22:09.400947 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:09.400792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" event={"ID":"7c16c068-807b-4d62-9829-5091e332e053","Type":"ContainerDied","Data":"f42f3afe29fa71f92512810a62fdfd53fa43c03c9943a01206a83f6fc317e5d2"} Jul 20 13:22:09.401071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:09.401059 2577 scope.go:117] "RemoveContainer" containerID="f42f3afe29fa71f92512810a62fdfd53fa43c03c9943a01206a83f6fc317e5d2" Jul 20 13:22:10.406232 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:10.406197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-g6pwt" event={"ID":"7c16c068-807b-4d62-9829-5091e332e053","Type":"ContainerStarted","Data":"34342041e8a9e2f35144697c11f2c1399fdbefa3fc672f75ada571a03dd6ed57"} Jul 20 13:22:24.024934 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:24.024895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:24.121371 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:24.121338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:28.448127 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:28.448094 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:22:28.451751 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:28.451730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7dfdcb75f-45zq8" Jul 20 13:22:29.470331 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:29.470299 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ab538b1-13e3-4709-90bb-05507ba05fcf" containerID="621b136f737bb3484a5f830a4121d249c8cafd445c46d626506bbc15b528c452" exitCode=0 Jul 20 13:22:29.470772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:29.470367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" event={"ID":"1ab538b1-13e3-4709-90bb-05507ba05fcf","Type":"ContainerDied","Data":"621b136f737bb3484a5f830a4121d249c8cafd445c46d626506bbc15b528c452"} Jul 20 13:22:29.470772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:29.470670 2577 scope.go:117] "RemoveContainer" containerID="621b136f737bb3484a5f830a4121d249c8cafd445c46d626506bbc15b528c452" Jul 20 13:22:30.475538 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:30.475504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-4v8hd" event={"ID":"1ab538b1-13e3-4709-90bb-05507ba05fcf","Type":"ContainerStarted","Data":"f391f185165d8415e0ed2adafe98cbfd3c057b75d4dfb2767af7008358691959"} Jul 20 13:22:35.284490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:35.284402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:35.378679 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:35.378646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:39.509252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:39.509215 2577 generic.go:356] "Generic (PLEG): container finished" podID="66e6e7e3-929b-4928-9e5b-d3ac05572fad" containerID="8b07a672ebbe9259e8170a322d157613b587d277c27ef926093664544d6d5e68" exitCode=0 Jul 20 13:22:39.509632 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:39.509257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" event={"ID":"66e6e7e3-929b-4928-9e5b-d3ac05572fad","Type":"ContainerDied","Data":"8b07a672ebbe9259e8170a322d157613b587d277c27ef926093664544d6d5e68"} Jul 20 13:22:39.509632 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:39.509544 2577 scope.go:117] "RemoveContainer" containerID="8b07a672ebbe9259e8170a322d157613b587d277c27ef926093664544d6d5e68" Jul 20 13:22:40.264277 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:40.264231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:40.331466 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:40.331436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:40.516573 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:40.516478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-bj7gj" event={"ID":"66e6e7e3-929b-4928-9e5b-d3ac05572fad","Type":"ContainerStarted","Data":"b977f18341d1327709b2b3b5c628bced5d8c518ace2ca2cd4b47e674e198c964"} Jul 20 13:22:55.298226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.298189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:22:55.300472 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.300452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11833f47-b5c7-4ad0-abf8-878cde93e3c9-metrics-certs\") pod \"network-metrics-daemon-52sdt\" (UID: \"11833f47-b5c7-4ad0-abf8-878cde93e3c9\") " pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:22:55.390217 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.390189 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:22:55.397874 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.397855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-52sdt" Jul 20 13:22:55.526529 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.526476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-52sdt"] Jul 20 13:22:55.528628 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:22:55.528603 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11833f47_b5c7_4ad0_abf8_878cde93e3c9.slice/crio-eaee124aa1ff6f59585f90a9e7a2bb7f738f2d4ac6dc962921d354b6507c67e9 WatchSource:0}: Error finding container eaee124aa1ff6f59585f90a9e7a2bb7f738f2d4ac6dc962921d354b6507c67e9: Status 404 returned error can't find the container with id eaee124aa1ff6f59585f90a9e7a2bb7f738f2d4ac6dc962921d354b6507c67e9 Jul 20 13:22:55.573272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:55.573210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52sdt" event={"ID":"11833f47-b5c7-4ad0-abf8-878cde93e3c9","Type":"ContainerStarted","Data":"eaee124aa1ff6f59585f90a9e7a2bb7f738f2d4ac6dc962921d354b6507c67e9"} Jul 20 13:22:57.374146 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.374116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:57.476291 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.476255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:57.582458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.582425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52sdt" event={"ID":"11833f47-b5c7-4ad0-abf8-878cde93e3c9","Type":"ContainerStarted","Data":"f1bdb6712bcd110237ffc209126f07cce4d9fc6ac16e697f6c63365b60f40da5"} Jul 20 13:22:57.582583 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.582487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-52sdt" event={"ID":"11833f47-b5c7-4ad0-abf8-878cde93e3c9","Type":"ContainerStarted","Data":"fa021bde6e0b6d36de4b1fe530e83a2a56c6819e5723d5413f229e0d210e4ef3"} Jul 20 13:22:57.601588 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.601538 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-52sdt" podStartSLOduration=375.543525879 podStartE2EDuration="6m16.601522444s" totalImagesPullingTime="1.057996565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:22:57.599246848 +0000 UTC m=+376.663595762" watchObservedRunningTime="2026-07-20 13:22:57.601522444 +0000 UTC m=+376.665871362" Jul 20 13:22:57.729662 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.729586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:22:57.798219 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:22:57.798186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:01.675564 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:01.675519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:01.756181 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:01.756150 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:03.086998 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:03.086968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:03.187711 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:03.187677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:12.163080 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:12.163047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:12.236272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:12.236240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:18.385602 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:18.385558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:18.453898 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:18.453864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:19.099164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:19.099131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:30.408363 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:30.408324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:30.513651 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:30.513624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:30.607829 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:30.607796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:30.708031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:30.707962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:30.787661 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:30.787631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:39.041711 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:39.041672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:23:39.135811 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:23:39.135775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:10.433805 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:10.433732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:10.512461 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:10.512425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:22.678946 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:22.678904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:33.230408 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:33.230374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:33.323356 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:33.323314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:34.006381 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.006350 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg"] Jul 20 13:24:34.006664 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.006644 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" containerName="registry" Jul 20 13:24:34.006718 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.006666 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" containerName="registry" Jul 20 13:24:34.006759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.006717 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b341a888-80bf-468f-b7c8-1864e6f1ec0e" containerName="registry" Jul 20 13:24:34.009704 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.009687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.013048 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.013026 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pm7s9\"" Jul 20 13:24:34.013357 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.013333 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 13:24:34.014010 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.013993 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 13:24:34.027849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.027824 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg"] Jul 20 13:24:34.142964 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.142926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.143147 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.142968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsmj\" (UniqueName: \"kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.143147 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.143000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.243528 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.243474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.243528 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.243530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsmj\" (UniqueName: \"kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.244053 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.243562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.244053 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.243939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.244053 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.243994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.252495 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.252471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsmj\" (UniqueName: \"kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.318261 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.318209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:34.445997 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.445907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg"] Jul 20 13:24:34.448954 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:24:34.448928 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b21848_814c_4920_b426_71d31b838a2b.slice/crio-75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef WatchSource:0}: Error finding container 75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef: Status 404 returned error can't find the container with id 75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef Jul 20 13:24:34.939842 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:34.939807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerStarted","Data":"75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef"} Jul 20 13:24:40.965855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:40.965819 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b21848-814c-4920-b426-71d31b838a2b" containerID="aaee3292624e4b6aa106d01cb2e359ab33caf593dfda4f74ba2a079c4d4d567e" exitCode=0 Jul 20 13:24:40.965855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:40.965857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerDied","Data":"aaee3292624e4b6aa106d01cb2e359ab33caf593dfda4f74ba2a079c4d4d567e"} Jul 20 13:24:42.976381 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:42.976344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerStarted","Data":"b0ae54f42345030a86a9ab2694d21bc08dd446dc2e2c3600347685f448b04841"} Jul 20 13:24:43.981456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:43.981419 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b21848-814c-4920-b426-71d31b838a2b" containerID="b0ae54f42345030a86a9ab2694d21bc08dd446dc2e2c3600347685f448b04841" exitCode=0 Jul 20 13:24:43.981849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:43.981501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerDied","Data":"b0ae54f42345030a86a9ab2694d21bc08dd446dc2e2c3600347685f448b04841"} Jul 20 13:24:47.668759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:47.668723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:51.013959 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:51.013921 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b21848-814c-4920-b426-71d31b838a2b" containerID="f717766962dc05199eae6f224121b8b16b15f1ce6c937cd27dca53f17ef652ee" exitCode=0 Jul 20 13:24:51.014368 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:51.013982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerDied","Data":"f717766962dc05199eae6f224121b8b16b15f1ce6c937cd27dca53f17ef652ee"} Jul 20 13:24:52.137489 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.137435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:52.304446 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.304416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util\") pod \"c2b21848-814c-4920-b426-71d31b838a2b\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " Jul 20 13:24:52.304446 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.304449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle\") pod \"c2b21848-814c-4920-b426-71d31b838a2b\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " Jul 20 13:24:52.304643 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.304505 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwsmj\" (UniqueName: \"kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj\") pod \"c2b21848-814c-4920-b426-71d31b838a2b\" (UID: \"c2b21848-814c-4920-b426-71d31b838a2b\") " Jul 20 13:24:52.305137 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.305106 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle" (OuterVolumeSpecName: "bundle") pod "c2b21848-814c-4920-b426-71d31b838a2b" (UID: "c2b21848-814c-4920-b426-71d31b838a2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:24:52.306635 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.306611 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj" (OuterVolumeSpecName: "kube-api-access-kwsmj") pod "c2b21848-814c-4920-b426-71d31b838a2b" (UID: "c2b21848-814c-4920-b426-71d31b838a2b"). InnerVolumeSpecName "kube-api-access-kwsmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:24:52.308725 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.308699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util" (OuterVolumeSpecName: "util") pod "c2b21848-814c-4920-b426-71d31b838a2b" (UID: "c2b21848-814c-4920-b426-71d31b838a2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:24:52.404957 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.404891 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsmj\" (UniqueName: \"kubernetes.io/projected/c2b21848-814c-4920-b426-71d31b838a2b-kube-api-access-kwsmj\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:24:52.404957 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.404916 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:24:52.404957 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:52.404929 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b21848-814c-4920-b426-71d31b838a2b-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:24:53.022797 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:53.022759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" event={"ID":"c2b21848-814c-4920-b426-71d31b838a2b","Type":"ContainerDied","Data":"75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef"} Jul 20 13:24:53.022797 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:53.022789 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540btrxg" Jul 20 13:24:53.022999 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:53.022806 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ad901ca4b42049de380f52df2cb7996e6195741c7e903d051d1116b3288eef" Jul 20 13:24:55.078664 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:55.078630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:56.071152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.071111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:56.986675 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986630 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7"] Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986911 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="pull" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986922 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="pull" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986937 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="util" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986942 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="util" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="extract" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.986954 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="extract" Jul 20 13:24:56.987064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.987010 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b21848-814c-4920-b426-71d31b838a2b" containerName="extract" Jul 20 13:24:56.990513 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.990496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:56.993222 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.993200 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:24:56.993742 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.993722 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 20 13:24:56.993799 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:56.993767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-56jhm\"" Jul 20 13:24:57.008748 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.008724 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7"] Jul 20 13:24:57.145593 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.145551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6vm\" (UniqueName: \"kubernetes.io/projected/ffb527f4-17f4-400f-ad3e-a25c4840eb13-kube-api-access-cn6vm\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.145593 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.145598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb527f4-17f4-400f-ad3e-a25c4840eb13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.246781 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.246691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6vm\" (UniqueName: \"kubernetes.io/projected/ffb527f4-17f4-400f-ad3e-a25c4840eb13-kube-api-access-cn6vm\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.246781 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.246735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb527f4-17f4-400f-ad3e-a25c4840eb13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.247153 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.247133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb527f4-17f4-400f-ad3e-a25c4840eb13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.258855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.258826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6vm\" (UniqueName: \"kubernetes.io/projected/ffb527f4-17f4-400f-ad3e-a25c4840eb13-kube-api-access-cn6vm\") pod \"cert-manager-operator-controller-manager-55c444bf86-rshn7\" (UID: \"ffb527f4-17f4-400f-ad3e-a25c4840eb13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.299535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.299491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" Jul 20 13:24:57.432445 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:57.432421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7"] Jul 20 13:24:57.435165 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:24:57.435134 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb527f4_17f4_400f_ad3e_a25c4840eb13.slice/crio-9fd48d10313f9728fbcd782dc7fab9ed24a86356026dd413f9bd7c84e077f03b WatchSource:0}: Error finding container 9fd48d10313f9728fbcd782dc7fab9ed24a86356026dd413f9bd7c84e077f03b: Status 404 returned error can't find the container with id 9fd48d10313f9728fbcd782dc7fab9ed24a86356026dd413f9bd7c84e077f03b Jul 20 13:24:58.044768 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:58.044733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" event={"ID":"ffb527f4-17f4-400f-ad3e-a25c4840eb13","Type":"ContainerStarted","Data":"9fd48d10313f9728fbcd782dc7fab9ed24a86356026dd413f9bd7c84e077f03b"} Jul 20 13:24:58.673816 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:58.673758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:58.777307 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:58.777236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:59.051417 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:59.051385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" event={"ID":"ffb527f4-17f4-400f-ad3e-a25c4840eb13","Type":"ContainerStarted","Data":"bc684382470be4c38d30bb2dceaa4da2b1b21804464751c3d82c292258a24f7e"} Jul 20 13:24:59.077054 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:59.076992 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rshn7" podStartSLOduration=1.563199964 podStartE2EDuration="3.076970332s" totalImagesPullingTime="1.513770368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:24:59.074967567 +0000 UTC m=+498.139316482" watchObservedRunningTime="2026-07-20 13:24:59.076970332 +0000 UTC m=+498.141319248" Jul 20 13:24:59.528974 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:59.528943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:59.685134 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:59.685099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:24:59.857971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:24:59.857897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:00.497630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.497594 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz"] Jul 20 13:25:00.501123 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.501103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.503428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.503397 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pm7s9\"" Jul 20 13:25:00.503428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.503407 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 13:25:00.504486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.504454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 13:25:00.510821 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.510799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz"] Jul 20 13:25:00.577991 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.577955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.578213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.578010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.578213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.578032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmtz\" (UniqueName: \"kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.679284 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.679242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.679478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.679320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.679478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.679354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmtz\" (UniqueName: \"kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.679741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.679702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.679812 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.679726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.698343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.698306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmtz\" (UniqueName: \"kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.811060 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.810979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:00.947086 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:00.947060 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz"] Jul 20 13:25:00.949083 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:00.949057 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c74df3_e278_4942_98a7_b134010aac05.slice/crio-01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007 WatchSource:0}: Error finding container 01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007: Status 404 returned error can't find the container with id 01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007 Jul 20 13:25:01.060468 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:01.060439 2577 generic.go:356] "Generic (PLEG): container finished" podID="94c74df3-e278-4942-98a7-b134010aac05" containerID="15484e6c6259aebd48cafb86cb4dcbd97663e3d5eafe496a80f002191292b57d" exitCode=0 Jul 20 13:25:01.060594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:01.060474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" event={"ID":"94c74df3-e278-4942-98a7-b134010aac05","Type":"ContainerDied","Data":"15484e6c6259aebd48cafb86cb4dcbd97663e3d5eafe496a80f002191292b57d"} Jul 20 13:25:01.060594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:01.060508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" event={"ID":"94c74df3-e278-4942-98a7-b134010aac05","Type":"ContainerStarted","Data":"01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007"} Jul 20 13:25:02.063978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.063886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:02.484808 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.484775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:02.499525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.499491 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zcms"] Jul 20 13:25:02.508262 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.508236 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.511020 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.510960 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g66m9\"" Jul 20 13:25:02.511020 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.511003 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 13:25:02.512729 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.512696 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 13:25:02.515029 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.515005 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zcms"] Jul 20 13:25:02.595533 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.595495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.595726 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.595547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z6p\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-kube-api-access-59z6p\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.696321 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.696279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.696509 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.696338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z6p\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-kube-api-access-59z6p\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.705558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.705530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.705837 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.705808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z6p\" (UniqueName: \"kubernetes.io/projected/0ee377d1-784d-4c57-a5e3-5fb1cedac92a-kube-api-access-59z6p\") pod \"cert-manager-cainjector-559cdd8c87-5zcms\" (UID: \"0ee377d1-784d-4c57-a5e3-5fb1cedac92a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:02.826945 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:02.826876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" Jul 20 13:25:03.007666 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:03.007637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zcms"] Jul 20 13:25:03.025046 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:03.025019 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee377d1_784d_4c57_a5e3_5fb1cedac92a.slice/crio-e9488de933592c28e4880362d9f91a7c263bc48423f5280f71b944ff1ba33a3a WatchSource:0}: Error finding container e9488de933592c28e4880362d9f91a7c263bc48423f5280f71b944ff1ba33a3a: Status 404 returned error can't find the container with id e9488de933592c28e4880362d9f91a7c263bc48423f5280f71b944ff1ba33a3a Jul 20 13:25:03.070298 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:03.070262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" event={"ID":"0ee377d1-784d-4c57-a5e3-5fb1cedac92a","Type":"ContainerStarted","Data":"e9488de933592c28e4880362d9f91a7c263bc48423f5280f71b944ff1ba33a3a"} Jul 20 13:25:03.072048 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:03.072024 2577 generic.go:356] "Generic (PLEG): container finished" podID="94c74df3-e278-4942-98a7-b134010aac05" containerID="d8465bc9d9c64843e988a56319666bdf54cfa0fb87428bb29e0dec38a77e2cf3" exitCode=0 Jul 20 13:25:03.072147 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:03.072063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" event={"ID":"94c74df3-e278-4942-98a7-b134010aac05","Type":"ContainerDied","Data":"d8465bc9d9c64843e988a56319666bdf54cfa0fb87428bb29e0dec38a77e2cf3"} Jul 20 13:25:04.079486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:04.079402 2577 generic.go:356] "Generic (PLEG): container finished" podID="94c74df3-e278-4942-98a7-b134010aac05" containerID="b8c7e8d3d2164ce3adb2d4c0a8e015d20f60497f4ae140d5f1dd1429b8f333bf" exitCode=0 Jul 20 13:25:04.079486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:04.079473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" event={"ID":"94c74df3-e278-4942-98a7-b134010aac05","Type":"ContainerDied","Data":"b8c7e8d3d2164ce3adb2d4c0a8e015d20f60497f4ae140d5f1dd1429b8f333bf"} Jul 20 13:25:05.282999 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.282977 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:05.315350 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.315319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:05.318858 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.318823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle\") pod \"94c74df3-e278-4942-98a7-b134010aac05\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " Jul 20 13:25:05.318986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.318952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util\") pod \"94c74df3-e278-4942-98a7-b134010aac05\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " Jul 20 13:25:05.319043 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.318985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cmtz\" (UniqueName: \"kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz\") pod \"94c74df3-e278-4942-98a7-b134010aac05\" (UID: \"94c74df3-e278-4942-98a7-b134010aac05\") " Jul 20 13:25:05.319322 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.319292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle" (OuterVolumeSpecName: "bundle") pod "94c74df3-e278-4942-98a7-b134010aac05" (UID: "94c74df3-e278-4942-98a7-b134010aac05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:25:05.321334 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.321302 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz" (OuterVolumeSpecName: "kube-api-access-4cmtz") pod "94c74df3-e278-4942-98a7-b134010aac05" (UID: "94c74df3-e278-4942-98a7-b134010aac05"). InnerVolumeSpecName "kube-api-access-4cmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:25:05.325044 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.325016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util" (OuterVolumeSpecName: "util") pod "94c74df3-e278-4942-98a7-b134010aac05" (UID: "94c74df3-e278-4942-98a7-b134010aac05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:25:05.420780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.420708 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:05.420780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.420744 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmtz\" (UniqueName: \"kubernetes.io/projected/94c74df3-e278-4942-98a7-b134010aac05-kube-api-access-4cmtz\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:05.420780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.420761 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c74df3-e278-4942-98a7-b134010aac05-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:05.437357 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:05.437329 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:06.089792 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:06.089760 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" Jul 20 13:25:06.089992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:06.089764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgplz" event={"ID":"94c74df3-e278-4942-98a7-b134010aac05","Type":"ContainerDied","Data":"01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007"} Jul 20 13:25:06.089992 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:06.089922 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01cd74a2dba6b1db62e117eb885d3b9e382d294294b46a786a73690e73598007" Jul 20 13:25:06.091389 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:06.091359 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" event={"ID":"0ee377d1-784d-4c57-a5e3-5fb1cedac92a","Type":"ContainerStarted","Data":"37599ddf7dc713906d9be4494bcd72f2f7d90c8e1e191d5b0a677a9e3b93bd21"} Jul 20 13:25:06.108802 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:06.108738 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zcms" podStartSLOduration=1.797811914 podStartE2EDuration="4.108719241s" totalImagesPullingTime="2.310907327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:06.108209418 +0000 UTC m=+505.172558335" watchObservedRunningTime="2026-07-20 13:25:06.108719241 +0000 UTC m=+505.173068156" Jul 20 13:25:09.466515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:09.466478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:10.061699 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061665 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bz2qk"] Jul 20 13:25:10.061961 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="pull" Jul 20 13:25:10.061961 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061962 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="pull" Jul 20 13:25:10.062042 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061976 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="util" Jul 20 13:25:10.062042 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061983 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="util" Jul 20 13:25:10.062042 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.061998 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="extract" Jul 20 13:25:10.062042 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.062004 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="extract" Jul 20 13:25:10.062159 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.062053 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c74df3-e278-4942-98a7-b134010aac05" containerName="extract" Jul 20 13:25:10.064958 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.064942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.067802 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.067770 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wnnlk\"" Jul 20 13:25:10.074071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.074051 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bz2qk"] Jul 20 13:25:10.156458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.156428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.156458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.156468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrbr\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-kube-api-access-kkrbr\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.257617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.257581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.257617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.257621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrbr\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-kube-api-access-kkrbr\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.266205 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.266165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.266363 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.266345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrbr\" (UniqueName: \"kubernetes.io/projected/1dc89712-0106-4475-ac12-d8e7150981ce-kube-api-access-kkrbr\") pod \"cert-manager-55b4d954d5-bz2qk\" (UID: \"1dc89712-0106-4475-ac12-d8e7150981ce\") " pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.374210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.374184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bz2qk" Jul 20 13:25:10.502288 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.502264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bz2qk"] Jul 20 13:25:10.504806 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:10.504781 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc89712_0106_4475_ac12_d8e7150981ce.slice/crio-8d292ca66f5a2e5217fe26a163a425c7c1ceef80455aab06fc9f26e6f8c265d8 WatchSource:0}: Error finding container 8d292ca66f5a2e5217fe26a163a425c7c1ceef80455aab06fc9f26e6f8c265d8: Status 404 returned error can't find the container with id 8d292ca66f5a2e5217fe26a163a425c7c1ceef80455aab06fc9f26e6f8c265d8 Jul 20 13:25:10.785571 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.785494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:10.882895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:10.882849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:11.113379 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:11.113345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bz2qk" event={"ID":"1dc89712-0106-4475-ac12-d8e7150981ce","Type":"ContainerStarted","Data":"c0eb1454f455020cc39faee2a19ba77ac1ca48bdb13ae86e3560adf1abc22bda"} Jul 20 13:25:11.113599 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:11.113388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bz2qk" event={"ID":"1dc89712-0106-4475-ac12-d8e7150981ce","Type":"ContainerStarted","Data":"8d292ca66f5a2e5217fe26a163a425c7c1ceef80455aab06fc9f26e6f8c265d8"} Jul 20 13:25:17.576268 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:17.576231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:17.669138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:17.669106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:19.259001 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.258967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:19.840367 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.840319 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bz2qk" podStartSLOduration=9.840302646 podStartE2EDuration="9.840302646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:11.130626674 +0000 UTC m=+510.194975590" watchObservedRunningTime="2026-07-20 13:25:19.840302646 +0000 UTC m=+518.904651563" Jul 20 13:25:19.840729 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.840715 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj"] Jul 20 13:25:19.881651 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.881615 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj"] Jul 20 13:25:19.881807 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.881717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:19.884564 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.884539 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pm7s9\"" Jul 20 13:25:19.884703 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.884593 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 13:25:19.885490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.885469 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 13:25:19.931721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.931689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:19.931851 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.931755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftgm\" (UniqueName: \"kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:19.931851 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:19.931819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.032954 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.032919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.033109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.032974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftgm\" (UniqueName: \"kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.033109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.033007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.033337 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.033318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.033393 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.033373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.046397 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.046374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftgm\" (UniqueName: \"kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.191027 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.190994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:20.345687 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:20.345666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj"] Jul 20 13:25:20.347640 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:20.347609 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53156ebd_3b84_4123_a5ee_d685dc174d38.slice/crio-73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe WatchSource:0}: Error finding container 73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe: Status 404 returned error can't find the container with id 73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe Jul 20 13:25:21.158316 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:21.158273 2577 generic.go:356] "Generic (PLEG): container finished" podID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerID="ad13a29b30f38d72e143c50f78b5b0720df46dac7774eb87ae27ef402762e132" exitCode=0 Jul 20 13:25:21.158480 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:21.158337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" event={"ID":"53156ebd-3b84-4123-a5ee-d685dc174d38","Type":"ContainerDied","Data":"ad13a29b30f38d72e143c50f78b5b0720df46dac7774eb87ae27ef402762e132"} Jul 20 13:25:21.158480 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:21.158424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" event={"ID":"53156ebd-3b84-4123-a5ee-d685dc174d38","Type":"ContainerStarted","Data":"73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe"} Jul 20 13:25:23.168855 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:23.168815 2577 generic.go:356] "Generic (PLEG): container finished" podID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerID="68235b9941f72bee21b68f4174e0500d79d0490a96d36621fa7eca04e52dcd78" exitCode=0 Jul 20 13:25:23.169271 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:23.168901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" event={"ID":"53156ebd-3b84-4123-a5ee-d685dc174d38","Type":"ContainerDied","Data":"68235b9941f72bee21b68f4174e0500d79d0490a96d36621fa7eca04e52dcd78"} Jul 20 13:25:24.175790 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:24.175758 2577 generic.go:356] "Generic (PLEG): container finished" podID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerID="a3733a54f049403b7feca5cabb4fc3f49438ef355f496840d7924d5f5926e36b" exitCode=0 Jul 20 13:25:24.176138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:24.175821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" event={"ID":"53156ebd-3b84-4123-a5ee-d685dc174d38","Type":"ContainerDied","Data":"a3733a54f049403b7feca5cabb4fc3f49438ef355f496840d7924d5f5926e36b"} Jul 20 13:25:25.303801 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.303774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:25.377185 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.377149 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle\") pod \"53156ebd-3b84-4123-a5ee-d685dc174d38\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " Jul 20 13:25:25.377327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.377206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kftgm\" (UniqueName: \"kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm\") pod \"53156ebd-3b84-4123-a5ee-d685dc174d38\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " Jul 20 13:25:25.377327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.377247 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util\") pod \"53156ebd-3b84-4123-a5ee-d685dc174d38\" (UID: \"53156ebd-3b84-4123-a5ee-d685dc174d38\") " Jul 20 13:25:25.377841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.377810 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle" (OuterVolumeSpecName: "bundle") pod "53156ebd-3b84-4123-a5ee-d685dc174d38" (UID: "53156ebd-3b84-4123-a5ee-d685dc174d38"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:25:25.379336 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.379311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm" (OuterVolumeSpecName: "kube-api-access-kftgm") pod "53156ebd-3b84-4123-a5ee-d685dc174d38" (UID: "53156ebd-3b84-4123-a5ee-d685dc174d38"). InnerVolumeSpecName "kube-api-access-kftgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:25:25.382706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.382678 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util" (OuterVolumeSpecName: "util") pod "53156ebd-3b84-4123-a5ee-d685dc174d38" (UID: "53156ebd-3b84-4123-a5ee-d685dc174d38"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:25:25.478721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.478618 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:25.478721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.478647 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftgm\" (UniqueName: \"kubernetes.io/projected/53156ebd-3b84-4123-a5ee-d685dc174d38-kube-api-access-kftgm\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:25.478721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:25.478668 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53156ebd-3b84-4123-a5ee-d685dc174d38-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:25:26.185778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:26.185749 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" Jul 20 13:25:26.185778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:26.185772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxsjpj" event={"ID":"53156ebd-3b84-4123-a5ee-d685dc174d38","Type":"ContainerDied","Data":"73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe"} Jul 20 13:25:26.185978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:26.185811 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73471969657c622c43196c286c8c47c9fd169ac1adf66da6693691efe679a1fe" Jul 20 13:25:27.094657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:27.094614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:29.131512 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:29.131480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:34.062340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:34.062264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:34.163233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:34.163199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:34.732385 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:34.732349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.224573 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.224547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.325212 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.325165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.370345 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370310 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp"] Jul 20 13:25:35.370677 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370661 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="pull" Jul 20 13:25:35.370756 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370679 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="pull" Jul 20 13:25:35.370756 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370697 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="util" Jul 20 13:25:35.370756 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370705 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="util" Jul 20 13:25:35.370756 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370719 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="extract" Jul 20 13:25:35.370756 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370729 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="extract" Jul 20 13:25:35.370994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.370822 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53156ebd-3b84-4123-a5ee-d685dc174d38" containerName="extract" Jul 20 13:25:35.373882 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.373863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.376513 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.376486 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 13:25:35.376642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.376575 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 13:25:35.376757 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.376718 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 13:25:35.376832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.376768 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 13:25:35.376898 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.376853 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ggvq6\"" Jul 20 13:25:35.392841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.392812 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp"] Jul 20 13:25:35.464468 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.464444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.464602 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.464566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.464602 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.464598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.464681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.464617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.464681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.464637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgdf\" (UniqueName: \"kubernetes.io/projected/9e444982-5266-4908-9b51-6069e40d502c-kube-api-access-mvgdf\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.543773 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.543698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.565813 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.565791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.565892 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.565821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.565892 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.565839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.565892 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.565859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgdf\" (UniqueName: \"kubernetes.io/projected/9e444982-5266-4908-9b51-6069e40d502c-kube-api-access-mvgdf\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.566044 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.565904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.566414 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.566387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.568080 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.568059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e444982-5266-4908-9b51-6069e40d502c-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.568481 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.568460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.568575 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.568553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e444982-5266-4908-9b51-6069e40d502c-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.574279 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.574256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgdf\" (UniqueName: \"kubernetes.io/projected/9e444982-5266-4908-9b51-6069e40d502c-kube-api-access-mvgdf\") pod \"opendatahub-operator-controller-manager-864d7fc49d-s7hvp\" (UID: \"9e444982-5266-4908-9b51-6069e40d502c\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.644319 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.644286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.684190 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.684136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:35.786069 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.786036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.833369 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.833350 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp"] Jul 20 13:25:35.835236 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:35.835210 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e444982_5266_4908_9b51_6069e40d502c.slice/crio-b570d332655b02ebc3f1b37a90192df6d1117d71ca0739c62e1cbb6b5d3dc559 WatchSource:0}: Error finding container b570d332655b02ebc3f1b37a90192df6d1117d71ca0739c62e1cbb6b5d3dc559: Status 404 returned error can't find the container with id b570d332655b02ebc3f1b37a90192df6d1117d71ca0739c62e1cbb6b5d3dc559 Jul 20 13:25:35.876087 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.876060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:35.966065 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:35.966030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:36.067514 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:36.067484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:36.229296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:36.229262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" event={"ID":"9e444982-5266-4908-9b51-6069e40d502c","Type":"ContainerStarted","Data":"b570d332655b02ebc3f1b37a90192df6d1117d71ca0739c62e1cbb6b5d3dc559"} Jul 20 13:25:36.589680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:36.589604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:36.710982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:36.710878 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:39.246483 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:39.246399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" event={"ID":"9e444982-5266-4908-9b51-6069e40d502c","Type":"ContainerStarted","Data":"947914768c949ef02426326b5f62ee000d2a9f678cae99018b18723c5e45eec7"} Jul 20 13:25:40.252043 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:40.251955 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e444982-5266-4908-9b51-6069e40d502c" containerID="947914768c949ef02426326b5f62ee000d2a9f678cae99018b18723c5e45eec7" exitCode=0 Jul 20 13:25:40.252043 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:40.252024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" event={"ID":"9e444982-5266-4908-9b51-6069e40d502c","Type":"ContainerDied","Data":"947914768c949ef02426326b5f62ee000d2a9f678cae99018b18723c5e45eec7"} Jul 20 13:25:41.257983 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:41.257948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" event={"ID":"9e444982-5266-4908-9b51-6069e40d502c","Type":"ContainerStarted","Data":"dd1f700ea778e86b5decd04b8ce1486f83090d7903ef807f378692a485c7b761"} Jul 20 13:25:41.258433 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:41.258038 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:41.290553 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:41.290505 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" podStartSLOduration=1.707409993 podStartE2EDuration="6.290490481s" totalImagesPullingTime="3.52295469s" totalInitContainerRuntime="1.060125798s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:41.290254628 +0000 UTC m=+540.354603543" watchObservedRunningTime="2026-07-20 13:25:41.290490481 +0000 UTC m=+540.354839396" Jul 20 13:25:41.595525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:41.595438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:41.692056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:41.692023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:52.264601 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:52.264526 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-s7hvp" Jul 20 13:25:54.716912 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:54.716879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:54.840392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:54.840358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:55.561852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.561812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:55.712727 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.712692 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q"] Jul 20 13:25:55.724670 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.724631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.727385 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.727358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 13:25:55.727536 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.727368 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 13:25:55.728031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.728016 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 13:25:55.729039 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.729020 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pmk9l\"" Jul 20 13:25:55.734569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.734550 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q"] Jul 20 13:25:55.738459 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.738439 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 13:25:55.842340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.842222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fe0dae-c906-4370-87ad-da63eb63a6f5-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.842340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.842264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwptk\" (UniqueName: \"kubernetes.io/projected/37fe0dae-c906-4370-87ad-da63eb63a6f5-kube-api-access-lwptk\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.842340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.842296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37fe0dae-c906-4370-87ad-da63eb63a6f5-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.900360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.900319 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mpzl2"] Jul 20 13:25:55.904110 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.904079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:55.908512 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.908490 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 20 13:25:55.908646 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.908533 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mfs2r\"" Jul 20 13:25:55.929472 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.929445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mpzl2"] Jul 20 13:25:55.943657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.943624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fe0dae-c906-4370-87ad-da63eb63a6f5-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.943657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.943657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwptk\" (UniqueName: \"kubernetes.io/projected/37fe0dae-c906-4370-87ad-da63eb63a6f5-kube-api-access-lwptk\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.943848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.943677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37fe0dae-c906-4370-87ad-da63eb63a6f5-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.945922 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.945890 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37fe0dae-c906-4370-87ad-da63eb63a6f5-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.946112 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.946094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fe0dae-c906-4370-87ad-da63eb63a6f5-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:55.951275 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.951245 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:55.955674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:55.955649 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwptk\" (UniqueName: \"kubernetes.io/projected/37fe0dae-c906-4370-87ad-da63eb63a6f5-kube-api-access-lwptk\") pod \"kube-auth-proxy-5c5c8fcf86-l6j7q\" (UID: \"37fe0dae-c906-4370-87ad-da63eb63a6f5\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:56.036128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.036093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" Jul 20 13:25:56.044663 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.044634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txgr\" (UniqueName: \"kubernetes.io/projected/9c771701-0b5e-46c5-ade8-6fca67cdec69-kube-api-access-2txgr\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.044772 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.044705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c771701-0b5e-46c5-ade8-6fca67cdec69-cert\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.057518 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.057480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:56.145688 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.145653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txgr\" (UniqueName: \"kubernetes.io/projected/9c771701-0b5e-46c5-ade8-6fca67cdec69-kube-api-access-2txgr\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.145938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.145753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c771701-0b5e-46c5-ade8-6fca67cdec69-cert\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.148410 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.148354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c771701-0b5e-46c5-ade8-6fca67cdec69-cert\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.160701 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.160657 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:56.165078 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.165056 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txgr\" (UniqueName: \"kubernetes.io/projected/9c771701-0b5e-46c5-ade8-6fca67cdec69-kube-api-access-2txgr\") pod \"odh-model-controller-7998c67dc4-mpzl2\" (UID: \"9c771701-0b5e-46c5-ade8-6fca67cdec69\") " pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.177435 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.177414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q"] Jul 20 13:25:56.179630 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:56.179605 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fe0dae_c906_4370_87ad_da63eb63a6f5.slice/crio-fc49b057ecadb8b84f94624bcc92a63b8c6de7fd7e96daebb2fe89278c17d289 WatchSource:0}: Error finding container fc49b057ecadb8b84f94624bcc92a63b8c6de7fd7e96daebb2fe89278c17d289: Status 404 returned error can't find the container with id fc49b057ecadb8b84f94624bcc92a63b8c6de7fd7e96daebb2fe89278c17d289 Jul 20 13:25:56.218778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.218744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:25:56.272944 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.272909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:56.330039 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.330002 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" event={"ID":"37fe0dae-c906-4370-87ad-da63eb63a6f5","Type":"ContainerStarted","Data":"fc49b057ecadb8b84f94624bcc92a63b8c6de7fd7e96daebb2fe89278c17d289"} Jul 20 13:25:56.369122 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:56.369091 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mpzl2"] Jul 20 13:25:56.370963 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:56.370932 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c771701_0b5e_46c5_ade8_6fca67cdec69.slice/crio-2c34ffd6f5a8068d0129472ec797130068a1931e7af316ec453bb4bf0c55dc62 WatchSource:0}: Error finding container 2c34ffd6f5a8068d0129472ec797130068a1931e7af316ec453bb4bf0c55dc62: Status 404 returned error can't find the container with id 2c34ffd6f5a8068d0129472ec797130068a1931e7af316ec453bb4bf0c55dc62 Jul 20 13:25:57.009480 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.009444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:57.204215 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.204153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:57.338459 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.338371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" event={"ID":"9c771701-0b5e-46c5-ade8-6fca67cdec69","Type":"ContainerStarted","Data":"2c34ffd6f5a8068d0129472ec797130068a1931e7af316ec453bb4bf0c55dc62"} Jul 20 13:25:57.522344 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.522303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:57.527436 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.527408 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:25:57.532042 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.532017 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.548478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.548454 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 13:25:57.549317 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.549299 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 13:25:57.549895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.549662 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s4zk9\"" Jul 20 13:25:57.549895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.549715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 13:25:57.550106 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.550082 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 13:25:57.561515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr72\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.561865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.561776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.576141 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.576106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:25:57.663056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr72\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.663943 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.663918 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.666510 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.666483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.666866 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.666842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.667119 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.667049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.667342 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.667323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.675774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.675745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.695228 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.695139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr72\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72\") pod \"istiod-openshift-gateway-7574459656-fbcs7\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:57.765418 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.765379 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:25:57.845125 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:57.845053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:25:58.042752 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:58.042720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:25:58.046504 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:25:58.046471 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0f817a_3aba_4f3e_b278_d5be0c11deb8.slice/crio-37aae4e32fb303fdc9bcdf7544372310e8c5e8127e4938861305e4f0a9d77cf6 WatchSource:0}: Error finding container 37aae4e32fb303fdc9bcdf7544372310e8c5e8127e4938861305e4f0a9d77cf6: Status 404 returned error can't find the container with id 37aae4e32fb303fdc9bcdf7544372310e8c5e8127e4938861305e4f0a9d77cf6 Jul 20 13:25:58.345897 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:58.345821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" event={"ID":"6c0f817a-3aba-4f3e-b278-d5be0c11deb8","Type":"ContainerStarted","Data":"37aae4e32fb303fdc9bcdf7544372310e8c5e8127e4938861305e4f0a9d77cf6"} Jul 20 13:25:59.424453 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:25:59.424421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:01.372797 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.372752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" event={"ID":"37fe0dae-c906-4370-87ad-da63eb63a6f5","Type":"ContainerStarted","Data":"de06c9dbdacd791b6a093a079eb2d5a6b457916477957e817aa48cc2bf31c012"} Jul 20 13:26:01.374586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.374555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" event={"ID":"9c771701-0b5e-46c5-ade8-6fca67cdec69","Type":"ContainerStarted","Data":"914e3c57d0c5d0f943a948c27f70813ca8af746330d86f3eefa102454e790729"} Jul 20 13:26:01.374730 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.374630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:26:01.398211 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.398065 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-l6j7q" podStartSLOduration=2.331645506 podStartE2EDuration="6.398044797s" totalImagesPullingTime="4.066399291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:01.395860004 +0000 UTC m=+560.460208921" watchObservedRunningTime="2026-07-20 13:26:01.398044797 +0000 UTC m=+560.462393714" Jul 20 13:26:01.415687 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.415624 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" podStartSLOduration=2.496921559 podStartE2EDuration="6.41560531s" totalImagesPullingTime="3.918683751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:01.415104942 +0000 UTC m=+560.479453858" watchObservedRunningTime="2026-07-20 13:26:01.41560531 +0000 UTC m=+560.479954225" Jul 20 13:26:01.637082 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.637046 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:26:01.637188 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:01.637114 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:26:02.382664 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.382616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" event={"ID":"6c0f817a-3aba-4f3e-b278-d5be0c11deb8","Type":"ContainerStarted","Data":"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1"} Jul 20 13:26:02.413986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.413949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:02.437330 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.437264 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" podStartSLOduration=1.850222493 podStartE2EDuration="5.437246922s" totalImagesPullingTime="3.587024429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:02.434283335 +0000 UTC m=+561.498632306" watchObservedRunningTime="2026-07-20 13:26:02.437246922 +0000 UTC m=+561.501595828" Jul 20 13:26:02.520284 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.520244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:02.855863 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.855752 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:02.996162 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:02.996125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:03.164214 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:03.164154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:03.392339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:03.392314 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:26:03.392697 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:03.392388 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:26:06.775893 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.775858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:06.873990 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.873954 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj"] Jul 20 13:26:06.877300 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.877284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:06.888541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.888518 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dgsgx\"" Jul 20 13:26:06.889208 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.889193 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 20 13:26:06.913665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.913632 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c"] Jul 20 13:26:06.917025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.917000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:06.930846 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.930820 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 20 13:26:06.931026 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.930884 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2qhks\"" Jul 20 13:26:06.949700 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.949662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:06.949854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.949732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49dx\" (UniqueName: \"kubernetes.io/projected/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-kube-api-access-t49dx\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:06.949854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.949760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:06.949948 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.949852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hc4\" (UniqueName: \"kubernetes.io/projected/a997532e-f76a-49b2-8116-79f131379202-kube-api-access-l9hc4\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:06.982008 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.981976 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj"] Jul 20 13:26:06.995911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:06.995878 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:07.011040 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.011005 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c"] Jul 20 13:26:07.050993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.050902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49dx\" (UniqueName: \"kubernetes.io/projected/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-kube-api-access-t49dx\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:07.050993 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.050949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.051245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.050997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hc4\" (UniqueName: \"kubernetes.io/projected/a997532e-f76a-49b2-8116-79f131379202-kube-api-access-l9hc4\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.051245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.051019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:07.051245 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:26:07.051129 2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 20 13:26:07.051245 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:26:07.051224 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert podName:a997532e-f76a-49b2-8116-79f131379202 nodeName:}" failed. No retries permitted until 2026-07-20 13:26:07.551199279 +0000 UTC m=+566.615548186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert") pod "llmisvc-controller-manager-777c6bd7d5-xbz8c" (UID: "a997532e-f76a-49b2-8116-79f131379202") : secret "llmisvc-webhook-server-cert" not found Jul 20 13:26:07.053389 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.053368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:07.077109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.077077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hc4\" (UniqueName: \"kubernetes.io/projected/a997532e-f76a-49b2-8116-79f131379202-kube-api-access-l9hc4\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.080969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.080940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49dx\" (UniqueName: \"kubernetes.io/projected/cb4adb05-d8d5-4fad-89ed-785b7a36e3c0-kube-api-access-t49dx\") pod \"kserve-controller-manager-d6f4b65b8-d7vqj\" (UID: \"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:07.152459 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.152423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:07.187692 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.187653 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:07.377196 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.377136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:07.400552 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.400405 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj"] Jul 20 13:26:07.403510 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:26:07.403466 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4adb05_d8d5_4fad_89ed_785b7a36e3c0.slice/crio-686e1371b454f7fa6653ca57eca195d2c7a66f2f1f14792b2b325288e3a590c8 WatchSource:0}: Error finding container 686e1371b454f7fa6653ca57eca195d2c7a66f2f1f14792b2b325288e3a590c8: Status 404 returned error can't find the container with id 686e1371b454f7fa6653ca57eca195d2c7a66f2f1f14792b2b325288e3a590c8 Jul 20 13:26:07.422844 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.422812 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:26:07.423009 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.422989 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" podUID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" containerName="discovery" containerID="cri-o://e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1" gracePeriod=30 Jul 20 13:26:07.568796 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.568748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.571670 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.571613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a997532e-f76a-49b2-8116-79f131379202-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xbz8c\" (UID: \"a997532e-f76a-49b2-8116-79f131379202\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.669902 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.669871 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:26:07.771026 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.770994 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771254 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771254 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771077 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771572 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771475 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771572 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771515 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:26:07.771572 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgr72\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771765 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771615 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771765 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token\") pod \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\" (UID: \"6c0f817a-3aba-4f3e-b278-d5be0c11deb8\") " Jul 20 13:26:07.771952 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.771933 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-ca-configmap\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.774285 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774189 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:26:07.774384 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774285 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs" (OuterVolumeSpecName: "local-certs") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:26:07.774384 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774322 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:26:07.774384 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774345 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts" (OuterVolumeSpecName: "cacerts") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:26:07.774490 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774451 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72" (OuterVolumeSpecName: "kube-api-access-tgr72") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "kube-api-access-tgr72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:26:07.774666 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.774639 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6c0f817a-3aba-4f3e-b278-d5be0c11deb8" (UID: "6c0f817a-3aba-4f3e-b278-d5be0c11deb8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:26:07.826870 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.826837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:07.872432 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872398 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-kubeconfig\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.872432 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872429 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-csr-dns-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.872432 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872439 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-cacerts\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.872721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872449 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgr72\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-kube-api-access-tgr72\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.872721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872459 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-local-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:07.872721 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:07.872467 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0f817a-3aba-4f3e-b278-d5be0c11deb8-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:26:08.009594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.009561 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c"] Jul 20 13:26:08.012736 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:26:08.012702 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda997532e_f76a_49b2_8116_79f131379202.slice/crio-da345a762889522d3b85754fb3a9c11aa0f1afce5fb06d2040a6b148524feacc WatchSource:0}: Error finding container da345a762889522d3b85754fb3a9c11aa0f1afce5fb06d2040a6b148524feacc: Status 404 returned error can't find the container with id da345a762889522d3b85754fb3a9c11aa0f1afce5fb06d2040a6b148524feacc Jul 20 13:26:08.233370 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.233294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:08.414138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.414102 2577 generic.go:356] "Generic (PLEG): container finished" podID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" containerID="e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1" exitCode=0 Jul 20 13:26:08.414352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.414161 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" Jul 20 13:26:08.414352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.414220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" event={"ID":"6c0f817a-3aba-4f3e-b278-d5be0c11deb8","Type":"ContainerDied","Data":"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1"} Jul 20 13:26:08.414352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.414274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7" event={"ID":"6c0f817a-3aba-4f3e-b278-d5be0c11deb8","Type":"ContainerDied","Data":"37aae4e32fb303fdc9bcdf7544372310e8c5e8127e4938861305e4f0a9d77cf6"} Jul 20 13:26:08.414352 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.414302 2577 scope.go:117] "RemoveContainer" containerID="e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1" Jul 20 13:26:08.416059 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.415768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" event={"ID":"a997532e-f76a-49b2-8116-79f131379202","Type":"ContainerStarted","Data":"da345a762889522d3b85754fb3a9c11aa0f1afce5fb06d2040a6b148524feacc"} Jul 20 13:26:08.417328 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.417303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" event={"ID":"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0","Type":"ContainerStarted","Data":"686e1371b454f7fa6653ca57eca195d2c7a66f2f1f14792b2b325288e3a590c8"} Jul 20 13:26:08.425642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.425622 2577 scope.go:117] "RemoveContainer" containerID="e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1" Jul 20 13:26:08.425929 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:26:08.425904 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1\": container with ID starting with e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1 not found: ID does not exist" containerID="e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1" Jul 20 13:26:08.426020 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.425941 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1"} err="failed to get container status \"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1\": rpc error: code = NotFound desc = could not find container \"e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1\": container with ID starting with e99d6a68a5a7edd362090cc8a6241860679472049b5e5c626147decc0e9bfca1 not found: ID does not exist" Jul 20 13:26:08.427840 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.427815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:08.464078 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.464046 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:26:08.470240 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:08.470212 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fbcs7"] Jul 20 13:26:09.394824 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:09.394786 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" path="/var/lib/kubelet/pods/6c0f817a-3aba-4f3e-b278-d5be0c11deb8/volumes" Jul 20 13:26:10.431024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:10.430975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" event={"ID":"cb4adb05-d8d5-4fad-89ed-785b7a36e3c0","Type":"ContainerStarted","Data":"d6cffdf24dfce7faafb2bd2de656c8225602c287c5de0037aa33b92fc5fca3c1"} Jul 20 13:26:10.431492 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:10.431096 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:10.454685 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:10.454627 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" podStartSLOduration=1.661812584 podStartE2EDuration="4.454613653s" totalImagesPullingTime="2.792801069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:10.453852552 +0000 UTC m=+569.518201478" watchObservedRunningTime="2026-07-20 13:26:10.454613653 +0000 UTC m=+569.518962567" Jul 20 13:26:12.385742 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:12.385707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mpzl2" Jul 20 13:26:12.442331 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:12.442293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" event={"ID":"a997532e-f76a-49b2-8116-79f131379202","Type":"ContainerStarted","Data":"b40f354000c158d69a567c75798e14eb940e51f2bf252ac64ae88322f805d130"} Jul 20 13:26:12.442520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:12.442386 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:12.465191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:12.465111 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" podStartSLOduration=3.109850413 podStartE2EDuration="6.465090134s" totalImagesPullingTime="3.355239721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:12.465055442 +0000 UTC m=+571.529404357" watchObservedRunningTime="2026-07-20 13:26:12.465090134 +0000 UTC m=+571.529439052" Jul 20 13:26:41.440249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:41.440224 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d7vqj" Jul 20 13:26:43.449644 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:43.449615 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xbz8c" Jul 20 13:26:47.137317 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:47.137287 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:26:47.138156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:47.138135 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:26:47.143260 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:47.143241 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:26:47.144156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:47.144137 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:26:56.329010 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.328969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:56.404645 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.404608 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:26:56.404994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.404976 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" containerName="discovery" Jul 20 13:26:56.405070 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.404996 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" containerName="discovery" Jul 20 13:26:56.405130 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.405107 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0f817a-3aba-4f3e-b278-d5be0c11deb8" containerName="discovery" Jul 20 13:26:56.408106 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.408085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:26:56.414072 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.414049 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 13:26:56.414225 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.414133 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8qdsq\"" Jul 20 13:26:56.427353 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.427331 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 13:26:56.491428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.491393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx57\" (UniqueName: \"kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57\") pod \"kuadrant-operator-catalog-zh8hh\" (UID: \"d746ae02-c0a6-454e-84da-1973e3e0f754\") " pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:26:56.528852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.528825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:26:56.592770 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.592689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx57\" (UniqueName: \"kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57\") pod \"kuadrant-operator-catalog-zh8hh\" (UID: \"d746ae02-c0a6-454e-84da-1973e3e0f754\") " pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:26:56.623763 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.623732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx57\" (UniqueName: \"kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57\") pod \"kuadrant-operator-catalog-zh8hh\" (UID: \"d746ae02-c0a6-454e-84da-1973e3e0f754\") " pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:26:56.659560 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.659527 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:26:56.659830 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.659812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:26:56.806576 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.806548 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:26:56.808478 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:26:56.808449 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd746ae02_c0a6_454e_84da_1973e3e0f754.slice/crio-dc61f0fb14a4a574f03146582b34208fa7e48db7401a127a42a001546d1513c2 WatchSource:0}: Error finding container dc61f0fb14a4a574f03146582b34208fa7e48db7401a127a42a001546d1513c2: Status 404 returned error can't find the container with id dc61f0fb14a4a574f03146582b34208fa7e48db7401a127a42a001546d1513c2 Jul 20 13:26:56.810297 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.810279 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:26:56.840475 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.840444 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t575"] Jul 20 13:26:56.844808 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.844768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:26:56.855476 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.855451 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t575"] Jul 20 13:26:56.894883 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.894849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkln\" (UniqueName: \"kubernetes.io/projected/3d01e344-ed7d-4e7d-a28c-3b545b09b42c-kube-api-access-nrkln\") pod \"kuadrant-operator-catalog-7t575\" (UID: \"3d01e344-ed7d-4e7d-a28c-3b545b09b42c\") " pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:26:56.995983 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:56.995947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkln\" (UniqueName: \"kubernetes.io/projected/3d01e344-ed7d-4e7d-a28c-3b545b09b42c-kube-api-access-nrkln\") pod \"kuadrant-operator-catalog-7t575\" (UID: \"3d01e344-ed7d-4e7d-a28c-3b545b09b42c\") " pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:26:57.008544 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:57.008509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkln\" (UniqueName: \"kubernetes.io/projected/3d01e344-ed7d-4e7d-a28c-3b545b09b42c-kube-api-access-nrkln\") pod \"kuadrant-operator-catalog-7t575\" (UID: \"3d01e344-ed7d-4e7d-a28c-3b545b09b42c\") " pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:26:57.154303 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:57.154268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:26:57.293303 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:57.293131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t575"] Jul 20 13:26:57.295900 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:26:57.295875 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d01e344_ed7d_4e7d_a28c_3b545b09b42c.slice/crio-54d894c08fccd9f3167db562f5067177a1931bdb87df60905238e2edce2cc141 WatchSource:0}: Error finding container 54d894c08fccd9f3167db562f5067177a1931bdb87df60905238e2edce2cc141: Status 404 returned error can't find the container with id 54d894c08fccd9f3167db562f5067177a1931bdb87df60905238e2edce2cc141 Jul 20 13:26:57.653998 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:57.653956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" event={"ID":"d746ae02-c0a6-454e-84da-1973e3e0f754","Type":"ContainerStarted","Data":"dc61f0fb14a4a574f03146582b34208fa7e48db7401a127a42a001546d1513c2"} Jul 20 13:26:57.655432 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:57.655399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7t575" event={"ID":"3d01e344-ed7d-4e7d-a28c-3b545b09b42c","Type":"ContainerStarted","Data":"54d894c08fccd9f3167db562f5067177a1931bdb87df60905238e2edce2cc141"} Jul 20 13:26:58.351698 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:58.351665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:26:59.667550 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.667503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" event={"ID":"d746ae02-c0a6-454e-84da-1973e3e0f754","Type":"ContainerStarted","Data":"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066"} Jul 20 13:26:59.668000 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.667559 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" podUID="d746ae02-c0a6-454e-84da-1973e3e0f754" containerName="registry-server" containerID="cri-o://585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066" gracePeriod=2 Jul 20 13:26:59.668942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.668918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7t575" event={"ID":"3d01e344-ed7d-4e7d-a28c-3b545b09b42c","Type":"ContainerStarted","Data":"25891599cb552e41ec154bea625dd16eb8781ce1d7f275ea46fec80b57ce9827"} Jul 20 13:26:59.704304 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.704255 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" podStartSLOduration=1.701172782 podStartE2EDuration="3.704242304s" totalImagesPullingTime="2.003069522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:59.702672185 +0000 UTC m=+618.767021101" watchObservedRunningTime="2026-07-20 13:26:59.704242304 +0000 UTC m=+618.768591219" Jul 20 13:26:59.727556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.727500 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7t575" podStartSLOduration=2.210523383 podStartE2EDuration="3.727487069s" totalImagesPullingTime="1.516963686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:59.72500046 +0000 UTC m=+618.789349374" watchObservedRunningTime="2026-07-20 13:26:59.727487069 +0000 UTC m=+618.791835983" Jul 20 13:26:59.913000 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:26:59.912975 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:27:00.023148 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.023055 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrx57\" (UniqueName: \"kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57\") pod \"d746ae02-c0a6-454e-84da-1973e3e0f754\" (UID: \"d746ae02-c0a6-454e-84da-1973e3e0f754\") " Jul 20 13:27:00.025286 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.025256 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57" (OuterVolumeSpecName: "kube-api-access-hrx57") pod "d746ae02-c0a6-454e-84da-1973e3e0f754" (UID: "d746ae02-c0a6-454e-84da-1973e3e0f754"). InnerVolumeSpecName "kube-api-access-hrx57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:27:00.123743 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.123704 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrx57\" (UniqueName: \"kubernetes.io/projected/d746ae02-c0a6-454e-84da-1973e3e0f754-kube-api-access-hrx57\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:00.674432 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.674397 2577 generic.go:356] "Generic (PLEG): container finished" podID="d746ae02-c0a6-454e-84da-1973e3e0f754" containerID="585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066" exitCode=0 Jul 20 13:27:00.674913 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.674440 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" Jul 20 13:27:00.674913 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.674486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" event={"ID":"d746ae02-c0a6-454e-84da-1973e3e0f754","Type":"ContainerDied","Data":"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066"} Jul 20 13:27:00.674913 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.674529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh8hh" event={"ID":"d746ae02-c0a6-454e-84da-1973e3e0f754","Type":"ContainerDied","Data":"dc61f0fb14a4a574f03146582b34208fa7e48db7401a127a42a001546d1513c2"} Jul 20 13:27:00.674913 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.674548 2577 scope.go:117] "RemoveContainer" containerID="585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066" Jul 20 13:27:00.684515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.684499 2577 scope.go:117] "RemoveContainer" containerID="585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066" Jul 20 13:27:00.684761 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:00.684743 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066\": container with ID starting with 585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066 not found: ID does not exist" containerID="585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066" Jul 20 13:27:00.684761 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.684771 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066"} err="failed to get container status \"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066\": rpc error: code = NotFound desc = could not find container \"585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066\": container with ID starting with 585573289bfbb78d6513937bfec618c43ced7cdfa1005a9cc80768f500438066 not found: ID does not exist" Jul 20 13:27:00.699149 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.699059 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:27:00.704332 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:00.704311 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh8hh"] Jul 20 13:27:01.392166 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:01.392133 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d746ae02-c0a6-454e-84da-1973e3e0f754" path="/var/lib/kubelet/pods/d746ae02-c0a6-454e-84da-1973e3e0f754/volumes" Jul 20 13:27:07.154825 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:07.154718 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:27:07.154825 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:07.154799 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:27:07.177198 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:07.177159 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:27:07.729982 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:07.729956 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7t575" Jul 20 13:27:11.489602 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:11.489563 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:11.644480 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:11.644452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:12.125229 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.125196 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m"] Jul 20 13:27:12.125557 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.125545 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d746ae02-c0a6-454e-84da-1973e3e0f754" containerName="registry-server" Jul 20 13:27:12.125610 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.125559 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d746ae02-c0a6-454e-84da-1973e3e0f754" containerName="registry-server" Jul 20 13:27:12.125649 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.125620 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d746ae02-c0a6-454e-84da-1973e3e0f754" containerName="registry-server" Jul 20 13:27:12.128885 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.128869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.142182 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.142155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrxcc\"" Jul 20 13:27:12.159207 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.159182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m"] Jul 20 13:27:12.216720 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.216694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbk6w\" (UniqueName: \"kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.216840 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.216746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.216915 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.216861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.317482 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.317443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.317598 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.317492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbk6w\" (UniqueName: \"kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.317598 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.317537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.317813 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.317794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.317873 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.317854 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.329850 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.329821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbk6w\" (UniqueName: \"kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.437661 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.437597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:12.503650 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.502950 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z"] Jul 20 13:27:12.507857 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.507836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.521465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.521442 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z"] Jul 20 13:27:12.603226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.603134 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m"] Jul 20 13:27:12.606010 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:12.605984 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1740153_04ec_470f_8901_d5b982e9f38d.slice/crio-9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093 WatchSource:0}: Error finding container 9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093: Status 404 returned error can't find the container with id 9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093 Jul 20 13:27:12.619477 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.619454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xwh\" (UniqueName: \"kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.619578 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.619508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.619578 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.619558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.720891 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.720866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.720986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.720920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.720986 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.720965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xwh\" (UniqueName: \"kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.721244 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.721223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.721327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.721306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.730550 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.730530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xwh\" (UniqueName: \"kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.733562 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.733534 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1740153-04ec-470f-8901-d5b982e9f38d" containerID="9d425c4907de47f7aa7bd4e4c2a3e387239657ffba1d1325bb9a83b555f81341" exitCode=0 Jul 20 13:27:12.733646 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.733580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" event={"ID":"e1740153-04ec-470f-8901-d5b982e9f38d","Type":"ContainerDied","Data":"9d425c4907de47f7aa7bd4e4c2a3e387239657ffba1d1325bb9a83b555f81341"} Jul 20 13:27:12.733646 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.733614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" event={"ID":"e1740153-04ec-470f-8901-d5b982e9f38d","Type":"ContainerStarted","Data":"9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093"} Jul 20 13:27:12.818412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.818388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:12.901976 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.901945 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4"] Jul 20 13:27:12.906959 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.906933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:12.921525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.921496 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4"] Jul 20 13:27:12.962338 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:12.962315 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z"] Jul 20 13:27:12.964217 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:12.964193 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f143e2b_3d07_4158_b7c1_d28fe54db4f9.slice/crio-989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62 WatchSource:0}: Error finding container 989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62: Status 404 returned error can't find the container with id 989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62 Jul 20 13:27:13.023068 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.023039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.023268 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.023101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.023268 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.023230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9glh\" (UniqueName: \"kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.124548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.124517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9glh\" (UniqueName: \"kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.124704 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.124560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.124704 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.124587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.124904 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.124889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.124979 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.124960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.133869 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.133848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9glh\" (UniqueName: \"kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.219221 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.219135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:13.358465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.358437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4"] Jul 20 13:27:13.361113 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:13.361088 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52115695_a382_4a33_bd51_6ac29a7510d5.slice/crio-65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125 WatchSource:0}: Error finding container 65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125: Status 404 returned error can't find the container with id 65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125 Jul 20 13:27:13.505410 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.505341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss"] Jul 20 13:27:13.508821 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.508801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.519806 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.519782 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss"] Jul 20 13:27:13.630182 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.630140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.630315 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.630246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.630315 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.630285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4pr\" (UniqueName: \"kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.731713 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.731690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.731819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.731731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.731819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.731751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4pr\" (UniqueName: \"kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.732089 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.732067 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.732201 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.732160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.740895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.740873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4pr\" (UniqueName: \"kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.741380 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.741358 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1740153-04ec-470f-8901-d5b982e9f38d" containerID="7d32bf82f0ae804bdfed186024fec10edb822b3fa7771f00c7f95a12ca582847" exitCode=0 Jul 20 13:27:13.741454 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.741433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" event={"ID":"e1740153-04ec-470f-8901-d5b982e9f38d","Type":"ContainerDied","Data":"7d32bf82f0ae804bdfed186024fec10edb822b3fa7771f00c7f95a12ca582847"} Jul 20 13:27:13.742994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.742971 2577 generic.go:356] "Generic (PLEG): container finished" podID="52115695-a382-4a33-bd51-6ac29a7510d5" containerID="a0fcdc2434606009455028067e6b4ea26d5f3e2cf3c06804902ba6a3e7d164c7" exitCode=0 Jul 20 13:27:13.743074 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.743052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" event={"ID":"52115695-a382-4a33-bd51-6ac29a7510d5","Type":"ContainerDied","Data":"a0fcdc2434606009455028067e6b4ea26d5f3e2cf3c06804902ba6a3e7d164c7"} Jul 20 13:27:13.743120 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.743105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" event={"ID":"52115695-a382-4a33-bd51-6ac29a7510d5","Type":"ContainerStarted","Data":"65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125"} Jul 20 13:27:13.744607 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.744588 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerID="b5430b2f253d00bdf979546917540b95c353b02b4afb28ebb28adaa434867df8" exitCode=0 Jul 20 13:27:13.744700 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.744612 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" event={"ID":"8f143e2b-3d07-4158-b7c1-d28fe54db4f9","Type":"ContainerDied","Data":"b5430b2f253d00bdf979546917540b95c353b02b4afb28ebb28adaa434867df8"} Jul 20 13:27:13.744700 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.744637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" event={"ID":"8f143e2b-3d07-4158-b7c1-d28fe54db4f9","Type":"ContainerStarted","Data":"989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62"} Jul 20 13:27:13.818626 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.818608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:13.955676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:13.955651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss"] Jul 20 13:27:13.957574 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:13.957545 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24830256_b72b_430c_8747_3eba634d1d51.slice/crio-6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43 WatchSource:0}: Error finding container 6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43: Status 404 returned error can't find the container with id 6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43 Jul 20 13:27:14.751794 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.751764 2577 generic.go:356] "Generic (PLEG): container finished" podID="52115695-a382-4a33-bd51-6ac29a7510d5" containerID="ce31c05c61745b70165fb4e36ed4d6c0ca82946b18df04dc3ce59542f9ebd704" exitCode=0 Jul 20 13:27:14.752139 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.751847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" event={"ID":"52115695-a382-4a33-bd51-6ac29a7510d5","Type":"ContainerDied","Data":"ce31c05c61745b70165fb4e36ed4d6c0ca82946b18df04dc3ce59542f9ebd704"} Jul 20 13:27:14.753414 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.753390 2577 generic.go:356] "Generic (PLEG): container finished" podID="24830256-b72b-430c-8747-3eba634d1d51" containerID="1e4777e5151c7239b29a6d36386479872aacfa879558a23cf6138be00cd8b234" exitCode=0 Jul 20 13:27:14.753502 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.753480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" event={"ID":"24830256-b72b-430c-8747-3eba634d1d51","Type":"ContainerDied","Data":"1e4777e5151c7239b29a6d36386479872aacfa879558a23cf6138be00cd8b234"} Jul 20 13:27:14.753561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.753529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" event={"ID":"24830256-b72b-430c-8747-3eba634d1d51","Type":"ContainerStarted","Data":"6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43"} Jul 20 13:27:14.755340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.755315 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerID="b3f8763fdd340cd55cb0576522430fe545d89f5c699c50bba1cb26620ee0c9b9" exitCode=0 Jul 20 13:27:14.755465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.755340 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" event={"ID":"8f143e2b-3d07-4158-b7c1-d28fe54db4f9","Type":"ContainerDied","Data":"b3f8763fdd340cd55cb0576522430fe545d89f5c699c50bba1cb26620ee0c9b9"} Jul 20 13:27:14.757755 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.757736 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1740153-04ec-470f-8901-d5b982e9f38d" containerID="3bfb2c929cd6c621e9770513611e631eaaa0b0de0dfef26a4ade7b7376805b98" exitCode=0 Jul 20 13:27:14.757860 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:14.757811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" event={"ID":"e1740153-04ec-470f-8901-d5b982e9f38d","Type":"ContainerDied","Data":"3bfb2c929cd6c621e9770513611e631eaaa0b0de0dfef26a4ade7b7376805b98"} Jul 20 13:27:15.764728 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.764697 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerID="52961c05a4983225114d91471238fa089e3bf918cf969229dffbf7117dfe72af" exitCode=0 Jul 20 13:27:15.765127 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.764797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" event={"ID":"8f143e2b-3d07-4158-b7c1-d28fe54db4f9","Type":"ContainerDied","Data":"52961c05a4983225114d91471238fa089e3bf918cf969229dffbf7117dfe72af"} Jul 20 13:27:15.766843 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.766812 2577 generic.go:356] "Generic (PLEG): container finished" podID="52115695-a382-4a33-bd51-6ac29a7510d5" containerID="36dd264dacae3d6d3e4535de72b31828d5e6c233e0249b64da01d4c9b2b99963" exitCode=0 Jul 20 13:27:15.766966 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.766885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" event={"ID":"52115695-a382-4a33-bd51-6ac29a7510d5","Type":"ContainerDied","Data":"36dd264dacae3d6d3e4535de72b31828d5e6c233e0249b64da01d4c9b2b99963"} Jul 20 13:27:15.768518 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.768491 2577 generic.go:356] "Generic (PLEG): container finished" podID="24830256-b72b-430c-8747-3eba634d1d51" containerID="ffdf34973c32e253207b2af5a630f527737cee15e2047398ea0195a5a1755e3a" exitCode=0 Jul 20 13:27:15.768631 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.768575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" event={"ID":"24830256-b72b-430c-8747-3eba634d1d51","Type":"ContainerDied","Data":"ffdf34973c32e253207b2af5a630f527737cee15e2047398ea0195a5a1755e3a"} Jul 20 13:27:15.958795 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:15.958771 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:16.054862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.054780 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle\") pod \"e1740153-04ec-470f-8901-d5b982e9f38d\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " Jul 20 13:27:16.054862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.054854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util\") pod \"e1740153-04ec-470f-8901-d5b982e9f38d\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " Jul 20 13:27:16.055061 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.054893 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbk6w\" (UniqueName: \"kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w\") pod \"e1740153-04ec-470f-8901-d5b982e9f38d\" (UID: \"e1740153-04ec-470f-8901-d5b982e9f38d\") " Jul 20 13:27:16.055256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.055229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle" (OuterVolumeSpecName: "bundle") pod "e1740153-04ec-470f-8901-d5b982e9f38d" (UID: "e1740153-04ec-470f-8901-d5b982e9f38d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:16.056935 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.056909 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w" (OuterVolumeSpecName: "kube-api-access-gbk6w") pod "e1740153-04ec-470f-8901-d5b982e9f38d" (UID: "e1740153-04ec-470f-8901-d5b982e9f38d"). InnerVolumeSpecName "kube-api-access-gbk6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:27:16.060677 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.060654 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util" (OuterVolumeSpecName: "util") pod "e1740153-04ec-470f-8901-d5b982e9f38d" (UID: "e1740153-04ec-470f-8901-d5b982e9f38d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:16.156041 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.156016 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:16.156041 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.156038 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1740153-04ec-470f-8901-d5b982e9f38d-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:16.156185 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.156046 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbk6w\" (UniqueName: \"kubernetes.io/projected/e1740153-04ec-470f-8901-d5b982e9f38d-kube-api-access-gbk6w\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:16.775757 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.775727 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" Jul 20 13:27:16.776193 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.775756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m" event={"ID":"e1740153-04ec-470f-8901-d5b982e9f38d","Type":"ContainerDied","Data":"9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093"} Jul 20 13:27:16.776193 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.775812 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c8010e4e47eefcdc9f6a6f62a6154c672d53e98b9742e17fd4dd9adc7917093" Jul 20 13:27:16.778026 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.778003 2577 generic.go:356] "Generic (PLEG): container finished" podID="24830256-b72b-430c-8747-3eba634d1d51" containerID="fb247af06167e388d0fb710fccb3ebe00b63a726ac62e07ccf9302b760dc2c81" exitCode=0 Jul 20 13:27:16.778146 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.778092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" event={"ID":"24830256-b72b-430c-8747-3eba634d1d51","Type":"ContainerDied","Data":"fb247af06167e388d0fb710fccb3ebe00b63a726ac62e07ccf9302b760dc2c81"} Jul 20 13:27:16.920357 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.920335 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:16.947392 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:16.947368 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:17.063031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.062958 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2xwh\" (UniqueName: \"kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh\") pod \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " Jul 20 13:27:17.063031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.062989 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util\") pod \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " Jul 20 13:27:17.063031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063016 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util\") pod \"52115695-a382-4a33-bd51-6ac29a7510d5\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " Jul 20 13:27:17.063324 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9glh\" (UniqueName: \"kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh\") pod \"52115695-a382-4a33-bd51-6ac29a7510d5\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " Jul 20 13:27:17.063324 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063134 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle\") pod \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\" (UID: \"8f143e2b-3d07-4158-b7c1-d28fe54db4f9\") " Jul 20 13:27:17.063324 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063156 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle\") pod \"52115695-a382-4a33-bd51-6ac29a7510d5\" (UID: \"52115695-a382-4a33-bd51-6ac29a7510d5\") " Jul 20 13:27:17.063860 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063829 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle" (OuterVolumeSpecName: "bundle") pod "52115695-a382-4a33-bd51-6ac29a7510d5" (UID: "52115695-a382-4a33-bd51-6ac29a7510d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:17.063928 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.063864 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle" (OuterVolumeSpecName: "bundle") pod "8f143e2b-3d07-4158-b7c1-d28fe54db4f9" (UID: "8f143e2b-3d07-4158-b7c1-d28fe54db4f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:17.065210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.065166 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh" (OuterVolumeSpecName: "kube-api-access-h2xwh") pod "8f143e2b-3d07-4158-b7c1-d28fe54db4f9" (UID: "8f143e2b-3d07-4158-b7c1-d28fe54db4f9"). InnerVolumeSpecName "kube-api-access-h2xwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:27:17.065494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.065472 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh" (OuterVolumeSpecName: "kube-api-access-k9glh") pod "52115695-a382-4a33-bd51-6ac29a7510d5" (UID: "52115695-a382-4a33-bd51-6ac29a7510d5"). InnerVolumeSpecName "kube-api-access-k9glh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:27:17.068962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.068933 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util" (OuterVolumeSpecName: "util") pod "52115695-a382-4a33-bd51-6ac29a7510d5" (UID: "52115695-a382-4a33-bd51-6ac29a7510d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:17.069517 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.069499 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util" (OuterVolumeSpecName: "util") pod "8f143e2b-3d07-4158-b7c1-d28fe54db4f9" (UID: "8f143e2b-3d07-4158-b7c1-d28fe54db4f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:17.163985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.163964 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.163985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.163985 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9glh\" (UniqueName: \"kubernetes.io/projected/52115695-a382-4a33-bd51-6ac29a7510d5-kube-api-access-k9glh\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.164117 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.163996 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.164117 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.164006 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52115695-a382-4a33-bd51-6ac29a7510d5-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.164117 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.164013 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xwh\" (UniqueName: \"kubernetes.io/projected/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-kube-api-access-h2xwh\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.164117 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.164021 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f143e2b-3d07-4158-b7c1-d28fe54db4f9-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:17.786324 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.786290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" event={"ID":"52115695-a382-4a33-bd51-6ac29a7510d5","Type":"ContainerDied","Data":"65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125"} Jul 20 13:27:17.786745 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.786373 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65dea9aeae23d3bf4cfa623d107bde659e4d29c8e0525cbf4c66f279531f7125" Jul 20 13:27:17.786745 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.786297 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4" Jul 20 13:27:17.788529 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.788509 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" Jul 20 13:27:17.788671 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.788543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z" event={"ID":"8f143e2b-3d07-4158-b7c1-d28fe54db4f9","Type":"ContainerDied","Data":"989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62"} Jul 20 13:27:17.788785 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.788771 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="989105c7f0ec12391fbcb2e1b565bf8bbe04ade45463a7cf45432099404d2f62" Jul 20 13:27:17.921558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:17.921537 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:18.071813 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.071732 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw4pr\" (UniqueName: \"kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr\") pod \"24830256-b72b-430c-8747-3eba634d1d51\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " Jul 20 13:27:18.071981 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.071830 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util\") pod \"24830256-b72b-430c-8747-3eba634d1d51\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " Jul 20 13:27:18.071981 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.071885 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle\") pod \"24830256-b72b-430c-8747-3eba634d1d51\" (UID: \"24830256-b72b-430c-8747-3eba634d1d51\") " Jul 20 13:27:18.072357 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.072323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle" (OuterVolumeSpecName: "bundle") pod "24830256-b72b-430c-8747-3eba634d1d51" (UID: "24830256-b72b-430c-8747-3eba634d1d51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:18.073822 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.073798 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr" (OuterVolumeSpecName: "kube-api-access-pw4pr") pod "24830256-b72b-430c-8747-3eba634d1d51" (UID: "24830256-b72b-430c-8747-3eba634d1d51"). InnerVolumeSpecName "kube-api-access-pw4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:27:18.076888 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.076850 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util" (OuterVolumeSpecName: "util") pod "24830256-b72b-430c-8747-3eba634d1d51" (UID: "24830256-b72b-430c-8747-3eba634d1d51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:27:18.172827 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.172798 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4pr\" (UniqueName: \"kubernetes.io/projected/24830256-b72b-430c-8747-3eba634d1d51-kube-api-access-pw4pr\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:18.172827 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.172824 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-util\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:18.172967 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.172835 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24830256-b72b-430c-8747-3eba634d1d51-bundle\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:27:18.794859 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.794827 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" event={"ID":"24830256-b72b-430c-8747-3eba634d1d51","Type":"ContainerDied","Data":"6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43"} Jul 20 13:27:18.795249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.794874 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss" Jul 20 13:27:18.795249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:18.794896 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c05f1ef0f0e52c762e9e95fa44a554a55f78e40e744dfcdb691fe920abdce43" Jul 20 13:27:22.155229 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:22.155196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:24.584064 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584033 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd"] Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584388 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="extract" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584401 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="extract" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584410 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="util" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584416 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="util" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584428 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="util" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584433 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="util" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584440 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="extract" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584445 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="extract" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584453 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="util" Jul 20 13:27:24.584457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584458 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="util" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584465 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584470 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584477 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584482 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584490 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584495 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584501 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584507 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584513 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584518 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584525 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584530 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="pull" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584537 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="util" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584541 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="util" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584617 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1740153-04ec-470f-8901-d5b982e9f38d" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584625 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52115695-a382-4a33-bd51-6ac29a7510d5" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584632 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f143e2b-3d07-4158-b7c1-d28fe54db4f9" containerName="extract" Jul 20 13:27:24.584750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.584637 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24830256-b72b-430c-8747-3eba634d1d51" containerName="extract" Jul 20 13:27:24.587929 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.587912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:24.593313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.593286 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zd64p\"" Jul 20 13:27:24.615693 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.615671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd"] Jul 20 13:27:24.727991 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.727959 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nd9\" (UniqueName: \"kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9\") pod \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" (UID: \"dabeff46-28a8-49ed-959c-ad276f0c2b4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:24.828507 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.828479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nd9\" (UniqueName: \"kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9\") pod \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" (UID: \"dabeff46-28a8-49ed-959c-ad276f0c2b4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:24.848457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.848405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nd9\" (UniqueName: \"kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9\") pod \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" (UID: \"dabeff46-28a8-49ed-959c-ad276f0c2b4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:24.898330 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:24.898305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:25.040785 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:25.040761 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd"] Jul 20 13:27:25.042426 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:25.042397 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabeff46_28a8_49ed_959c_ad276f0c2b4c.slice/crio-b9cbba326cbe05d1386167ee905d2d9b20176b26d5e3d7d64247c0ba3ac3eec3 WatchSource:0}: Error finding container b9cbba326cbe05d1386167ee905d2d9b20176b26d5e3d7d64247c0ba3ac3eec3: Status 404 returned error can't find the container with id b9cbba326cbe05d1386167ee905d2d9b20176b26d5e3d7d64247c0ba3ac3eec3 Jul 20 13:27:25.831085 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:25.831051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" event={"ID":"dabeff46-28a8-49ed-959c-ad276f0c2b4c","Type":"ContainerStarted","Data":"b9cbba326cbe05d1386167ee905d2d9b20176b26d5e3d7d64247c0ba3ac3eec3"} Jul 20 13:27:25.959290 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:25.959240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:27.843412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:27.843377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" event={"ID":"dabeff46-28a8-49ed-959c-ad276f0c2b4c","Type":"ContainerStarted","Data":"413896160ccd9138728edd6d55a68a27f57e5111f1e64d178b3451128e585e4d"} Jul 20 13:27:27.843856 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:27.843449 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:27.875780 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:27.875726 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" podStartSLOduration=1.979725976 podStartE2EDuration="3.875710638s" totalImagesPullingTime="1.895984662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:27:27.872555054 +0000 UTC m=+646.936903978" watchObservedRunningTime="2026-07-20 13:27:27.875710638 +0000 UTC m=+646.940059553" Jul 20 13:27:27.993895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:27.993857 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t"] Jul 20 13:27:28.000740 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.000710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:28.003007 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.002978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:28.004849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.004829 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-r5vrb\"" Jul 20 13:27:28.007494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.007472 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 20 13:27:28.017531 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.017511 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t"] Jul 20 13:27:28.054899 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.054868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wcg\" (UniqueName: \"kubernetes.io/projected/e78e390f-ad30-4a70-9909-16e614d07cb6-kube-api-access-58wcg\") pod \"dns-operator-controller-manager-864dd78c4c-k778t\" (UID: \"e78e390f-ad30-4a70-9909-16e614d07cb6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:28.156296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.156262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wcg\" (UniqueName: \"kubernetes.io/projected/e78e390f-ad30-4a70-9909-16e614d07cb6-kube-api-access-58wcg\") pod \"dns-operator-controller-manager-864dd78c4c-k778t\" (UID: \"e78e390f-ad30-4a70-9909-16e614d07cb6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:28.170894 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.170875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wcg\" (UniqueName: \"kubernetes.io/projected/e78e390f-ad30-4a70-9909-16e614d07cb6-kube-api-access-58wcg\") pod \"dns-operator-controller-manager-864dd78c4c-k778t\" (UID: \"e78e390f-ad30-4a70-9909-16e614d07cb6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:28.311188 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.311140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:28.322462 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.322427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:28.463856 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.460849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t"] Jul 20 13:27:28.467808 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:28.467780 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78e390f_ad30_4a70_9909_16e614d07cb6.slice/crio-21b8a03e5391030f3ea22fb9af93f2c797186553713a2e49447902374c6fe53c WatchSource:0}: Error finding container 21b8a03e5391030f3ea22fb9af93f2c797186553713a2e49447902374c6fe53c: Status 404 returned error can't find the container with id 21b8a03e5391030f3ea22fb9af93f2c797186553713a2e49447902374c6fe53c Jul 20 13:27:28.850657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:28.850546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" event={"ID":"e78e390f-ad30-4a70-9909-16e614d07cb6","Type":"ContainerStarted","Data":"21b8a03e5391030f3ea22fb9af93f2c797186553713a2e49447902374c6fe53c"} Jul 20 13:27:29.057974 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:29.057945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:29.465124 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:29.465088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:29.718969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:29.718887 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:30.868910 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:30.868866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" event={"ID":"e78e390f-ad30-4a70-9909-16e614d07cb6","Type":"ContainerStarted","Data":"5de224d613c1d5b306dc4483a29dcb445e5f0707445077bfeb602b66163fbeef"} Jul 20 13:27:30.869290 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:30.868941 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:30.894472 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:30.894424 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" podStartSLOduration=2.075830298 podStartE2EDuration="3.8944067s" totalImagesPullingTime="1.818576402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:27:30.89235129 +0000 UTC m=+649.956700206" watchObservedRunningTime="2026-07-20 13:27:30.8944067 +0000 UTC m=+649.958755618" Jul 20 13:27:31.368621 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:31.368586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:31.564046 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:31.564018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:31.668246 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:31.668151 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:32.036186 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.036091 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh"] Jul 20 13:27:32.039613 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.039592 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:32.044547 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.044526 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6c5gb\"" Jul 20 13:27:32.066817 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.066794 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh"] Jul 20 13:27:32.093568 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.093544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ffk5\" (UniqueName: \"kubernetes.io/projected/f3ecd588-9f66-44f8-8644-55ea36a7ab61-kube-api-access-4ffk5\") pod \"authorino-operator-75cc8d5cb7-tgtzh\" (UID: \"f3ecd588-9f66-44f8-8644-55ea36a7ab61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:32.194526 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.194497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ffk5\" (UniqueName: \"kubernetes.io/projected/f3ecd588-9f66-44f8-8644-55ea36a7ab61-kube-api-access-4ffk5\") pod \"authorino-operator-75cc8d5cb7-tgtzh\" (UID: \"f3ecd588-9f66-44f8-8644-55ea36a7ab61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:32.218260 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.218231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ffk5\" (UniqueName: \"kubernetes.io/projected/f3ecd588-9f66-44f8-8644-55ea36a7ab61-kube-api-access-4ffk5\") pod \"authorino-operator-75cc8d5cb7-tgtzh\" (UID: \"f3ecd588-9f66-44f8-8644-55ea36a7ab61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:32.349765 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.349698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:32.493212 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.493155 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh"] Jul 20 13:27:32.499578 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:32.499550 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ecd588_9f66_44f8_8644_55ea36a7ab61.slice/crio-c6393f206ba351a4af531fd489662937801f9b0ed24a316984b06285324fa4b5 WatchSource:0}: Error finding container c6393f206ba351a4af531fd489662937801f9b0ed24a316984b06285324fa4b5: Status 404 returned error can't find the container with id c6393f206ba351a4af531fd489662937801f9b0ed24a316984b06285324fa4b5 Jul 20 13:27:32.880294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.880261 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" event={"ID":"f3ecd588-9f66-44f8-8644-55ea36a7ab61","Type":"ContainerStarted","Data":"c6393f206ba351a4af531fd489662937801f9b0ed24a316984b06285324fa4b5"} Jul 20 13:27:32.915937 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:32.915906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:33.732014 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:33.731980 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:35.901446 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:35.901414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" event={"ID":"f3ecd588-9f66-44f8-8644-55ea36a7ab61","Type":"ContainerStarted","Data":"4f721d4f1d444f815bca413a59dc4367451b6f7eff14b21f382a82f751d5d023"} Jul 20 13:27:35.901823 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:35.901562 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:35.924228 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:35.924165 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" podStartSLOduration=2.082448051 podStartE2EDuration="4.9241504s" totalImagesPullingTime="2.841702349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:27:35.921596691 +0000 UTC m=+654.985945606" watchObservedRunningTime="2026-07-20 13:27:35.9241504 +0000 UTC m=+654.988499355" Jul 20 13:27:36.796222 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:36.796166 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:38.265473 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.265437 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq"] Jul 20 13:27:38.268781 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.268762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.273377 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.273362 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vdzsh\"" Jul 20 13:27:38.301440 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.301419 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq"] Jul 20 13:27:38.349781 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.349758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.349900 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.349881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2nq\" (UniqueName: \"kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.450958 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.450933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2nq\" (UniqueName: \"kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.451050 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.450967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.451318 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.451301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.460086 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.460063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2nq\" (UniqueName: \"kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.578713 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.578633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:38.723674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.723605 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq"] Jul 20 13:27:38.724525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.724502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:38.725898 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:38.725869 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b166c68_54d0_470c_93d7_a8ad5d7f1b04.slice/crio-50a7ff7b7389b095b59ddf75d6d150013fdeb0195a14c3e7dc7b8478ebd5a7f6 WatchSource:0}: Error finding container 50a7ff7b7389b095b59ddf75d6d150013fdeb0195a14c3e7dc7b8478ebd5a7f6: Status 404 returned error can't find the container with id 50a7ff7b7389b095b59ddf75d6d150013fdeb0195a14c3e7dc7b8478ebd5a7f6 Jul 20 13:27:38.825615 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.825587 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:38.853034 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.852970 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:38.919141 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:38.919106 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" event={"ID":"5b166c68-54d0-470c-93d7-a8ad5d7f1b04","Type":"ContainerStarted","Data":"50a7ff7b7389b095b59ddf75d6d150013fdeb0195a14c3e7dc7b8478ebd5a7f6"} Jul 20 13:27:41.129734 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:41.129697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:41.240214 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:41.240156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:41.877126 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:41.877095 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-k778t" Jul 20 13:27:43.949991 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:43.949913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" event={"ID":"5b166c68-54d0-470c-93d7-a8ad5d7f1b04","Type":"ContainerStarted","Data":"2518c929c981977a02cfad6beaf2652c56870eea648c7692e62da3d665783174"} Jul 20 13:27:43.950385 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:43.950048 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:43.975617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:43.975570 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" podStartSLOduration=1.069492097 podStartE2EDuration="5.975554412s" totalImagesPullingTime="4.906062315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:27:43.974038113 +0000 UTC m=+663.038387030" watchObservedRunningTime="2026-07-20 13:27:43.975554412 +0000 UTC m=+663.039903326" Jul 20 13:27:46.437732 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.437700 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc"] Jul 20 13:27:46.443977 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.443955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.449481 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.449457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 20 13:27:46.449481 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.449474 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 20 13:27:46.449754 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.449740 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrxcc\"" Jul 20 13:27:46.456374 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.456344 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc"] Jul 20 13:27:46.486354 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.486307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:46.520050 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.520022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18f9562-fe7e-422a-a77f-b48868ad8989-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.520219 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.520074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18f9562-fe7e-422a-a77f-b48868ad8989-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.520219 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.520139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bw6\" (UniqueName: \"kubernetes.io/projected/f18f9562-fe7e-422a-a77f-b48868ad8989-kube-api-access-b5bw6\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.620851 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.620816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18f9562-fe7e-422a-a77f-b48868ad8989-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.621025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.620857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bw6\" (UniqueName: \"kubernetes.io/projected/f18f9562-fe7e-422a-a77f-b48868ad8989-kube-api-access-b5bw6\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.621025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.620893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18f9562-fe7e-422a-a77f-b48868ad8989-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.621550 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.621527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18f9562-fe7e-422a-a77f-b48868ad8989-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.623299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.623280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18f9562-fe7e-422a-a77f-b48868ad8989-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.633302 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.633280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bw6\" (UniqueName: \"kubernetes.io/projected/f18f9562-fe7e-422a-a77f-b48868ad8989-kube-api-access-b5bw6\") pod \"kuadrant-console-plugin-66b5774985-9scvc\" (UID: \"f18f9562-fe7e-422a-a77f-b48868ad8989\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.705931 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.705831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:46.752552 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.752525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" Jul 20 13:27:46.839358 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.839320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:46.904609 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:27:46.904578 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf18f9562_fe7e_422a_a77f_b48868ad8989.slice/crio-9dad2782e7dd878307233449ecad06a84ef8768bb62ea0c68b72a84925f39e63 WatchSource:0}: Error finding container 9dad2782e7dd878307233449ecad06a84ef8768bb62ea0c68b72a84925f39e63: Status 404 returned error can't find the container with id 9dad2782e7dd878307233449ecad06a84ef8768bb62ea0c68b72a84925f39e63 Jul 20 13:27:46.906200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.906153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc"] Jul 20 13:27:46.909570 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.909551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tgtzh" Jul 20 13:27:46.968770 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:46.968686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" event={"ID":"f18f9562-fe7e-422a-a77f-b48868ad8989","Type":"ContainerStarted","Data":"9dad2782e7dd878307233449ecad06a84ef8768bb62ea0c68b72a84925f39e63"} Jul 20 13:27:47.095152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:47.095123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:47.411284 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:47.411253 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:51.422974 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:51.422928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:51.704621 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:51.704539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:27:54.959888 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:54.959857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:57.400434 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.400403 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq"] Jul 20 13:27:57.400897 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.400640 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" containerName="manager" containerID="cri-o://2518c929c981977a02cfad6beaf2652c56870eea648c7692e62da3d665783174" gracePeriod=2 Jul 20 13:27:57.422714 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.422687 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq"] Jul 20 13:27:57.452069 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.452035 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd"] Jul 20 13:27:57.452326 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.452296 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerName="manager" containerID="cri-o://413896160ccd9138728edd6d55a68a27f57e5111f1e64d178b3451128e585e4d" gracePeriod=2 Jul 20 13:27:57.466106 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.465748 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:27:57.466753 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.466406 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" containerName="manager" Jul 20 13:27:57.466753 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.466430 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" containerName="manager" Jul 20 13:27:57.466753 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.466539 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" containerName="manager" Jul 20 13:27:57.473142 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:57.473101 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:57.473401 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.473382 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd"] Jul 20 13:27:57.473521 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.473506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.517182 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.516922 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:27:57.519735 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.519709 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs"] Jul 20 13:27:57.520407 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.520388 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerName="manager" Jul 20 13:27:57.520478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.520412 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerName="manager" Jul 20 13:27:57.520539 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.520525 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerName="manager" Jul 20 13:27:57.525927 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.525904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:27:57.548978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.548940 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs"] Jul 20 13:27:57.550000 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:57.549969 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:57.552081 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:57.552049 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:57.558478 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:57.558453 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:57.582946 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:57.582920 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:57.627150 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.627117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.627343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.627197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.627343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.627270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkk2t\" (UniqueName: \"kubernetes.io/projected/c8b6ba05-4af1-46f3-a522-f899de075c31-kube-api-access-lkk2t\") pod \"limitador-operator-controller-manager-859d7b78bd-ccwxs\" (UID: \"c8b6ba05-4af1-46f3-a522-f899de075c31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:27:57.728710 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.728578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.728710 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.728706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.728933 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.728755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkk2t\" (UniqueName: \"kubernetes.io/projected/c8b6ba05-4af1-46f3-a522-f899de075c31-kube-api-access-lkk2t\") pod \"limitador-operator-controller-manager-859d7b78bd-ccwxs\" (UID: \"c8b6ba05-4af1-46f3-a522-f899de075c31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:27:57.729127 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.729099 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.741351 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.741318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkk2t\" (UniqueName: \"kubernetes.io/projected/c8b6ba05-4af1-46f3-a522-f899de075c31-kube-api-access-lkk2t\") pod \"limitador-operator-controller-manager-859d7b78bd-ccwxs\" (UID: \"c8b6ba05-4af1-46f3-a522-f899de075c31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:27:57.741474 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.741387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-78mgj\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.867280 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.867244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:27:57.873515 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:57.873485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:27:58.445031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.444998 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:27:58.470567 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.470534 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:27:58.470749 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.470664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.474482 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:58.474413 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:27:58.504023 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:27:58.503988 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:27:58.636989 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.636939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.637166 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.637050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt8c\" (UniqueName: \"kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.738402 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.738323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.738402 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.738393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt8c\" (UniqueName: \"kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.738774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.738750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.750196 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.750134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt8c\" (UniqueName: \"kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c\") pod \"kuadrant-operator-controller-manager-86878c769f-lx6nq\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:58.786961 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:58.786931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:27:59.852387 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:27:59.852333 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 13:28:08.106558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.106521 2577 generic.go:356] "Generic (PLEG): container finished" podID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" containerID="413896160ccd9138728edd6d55a68a27f57e5111f1e64d178b3451128e585e4d" exitCode=0 Jul 20 13:28:08.109867 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.109700 2577 generic.go:356] "Generic (PLEG): container finished" podID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" containerID="2518c929c981977a02cfad6beaf2652c56870eea648c7692e62da3d665783174" exitCode=0 Jul 20 13:28:08.119207 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.119185 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:08.121760 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:08.121728 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:08.155633 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.155609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:08.157890 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:08.157864 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:08.160029 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:08.160002 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:08.168842 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.168820 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:28:08.169105 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:08.169083 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f822d67_92f4_4392_8fb4_b16b319902c6.slice/crio-48e87a27bc193fca644f4ce1f63a5e013f7275778a4bcb017f24d86a008a9be3 WatchSource:0}: Error finding container 48e87a27bc193fca644f4ce1f63a5e013f7275778a4bcb017f24d86a008a9be3: Status 404 returned error can't find the container with id 48e87a27bc193fca644f4ce1f63a5e013f7275778a4bcb017f24d86a008a9be3 Jul 20 13:28:08.230281 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.230260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr2nq\" (UniqueName: \"kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq\") pod \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " Jul 20 13:28:08.230442 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.230431 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume\") pod \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\" (UID: \"5b166c68-54d0-470c-93d7-a8ad5d7f1b04\") " Jul 20 13:28:08.230853 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.230828 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5b166c68-54d0-470c-93d7-a8ad5d7f1b04" (UID: "5b166c68-54d0-470c-93d7-a8ad5d7f1b04"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:28:08.232021 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.231995 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq" (OuterVolumeSpecName: "kube-api-access-qr2nq") pod "5b166c68-54d0-470c-93d7-a8ad5d7f1b04" (UID: "5b166c68-54d0-470c-93d7-a8ad5d7f1b04"). InnerVolumeSpecName "kube-api-access-qr2nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:08.331033 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.331002 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2nd9\" (UniqueName: \"kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9\") pod \"dabeff46-28a8-49ed-959c-ad276f0c2b4c\" (UID: \"dabeff46-28a8-49ed-959c-ad276f0c2b4c\") " Jul 20 13:28:08.331381 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.331360 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2nq\" (UniqueName: \"kubernetes.io/projected/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-kube-api-access-qr2nq\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:08.331428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.331387 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b166c68-54d0-470c-93d7-a8ad5d7f1b04-extensions-socket-volume\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:08.332941 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.332923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9" (OuterVolumeSpecName: "kube-api-access-f2nd9") pod "dabeff46-28a8-49ed-959c-ad276f0c2b4c" (UID: "dabeff46-28a8-49ed-959c-ad276f0c2b4c"). InnerVolumeSpecName "kube-api-access-f2nd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:08.344088 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.344068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs"] Jul 20 13:28:08.345913 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:08.345888 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b6ba05_4af1_46f3_a522_f899de075c31.slice/crio-82a7051694add52290e1d7d0fc9637fa009368f07d33a3016664479a63fdc430 WatchSource:0}: Error finding container 82a7051694add52290e1d7d0fc9637fa009368f07d33a3016664479a63fdc430: Status 404 returned error can't find the container with id 82a7051694add52290e1d7d0fc9637fa009368f07d33a3016664479a63fdc430 Jul 20 13:28:08.346256 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.346240 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:28:08.353337 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:08.353279 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea130e8_163c_45be_89fa_563f771976fc.slice/crio-c99e0fe694f50cbba0c5e052545287d282515e374b11f1bad89934ff95956146 WatchSource:0}: Error finding container c99e0fe694f50cbba0c5e052545287d282515e374b11f1bad89934ff95956146: Status 404 returned error can't find the container with id c99e0fe694f50cbba0c5e052545287d282515e374b11f1bad89934ff95956146 Jul 20 13:28:08.431943 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:08.431891 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2nd9\" (UniqueName: \"kubernetes.io/projected/dabeff46-28a8-49ed-959c-ad276f0c2b4c-kube-api-access-f2nd9\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:09.118716 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.118685 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:09.118716 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.118699 2577 scope.go:117] "RemoveContainer" containerID="413896160ccd9138728edd6d55a68a27f57e5111f1e64d178b3451128e585e4d" Jul 20 13:28:09.120654 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.120619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" event={"ID":"0f822d67-92f4-4392-8fb4-b16b319902c6","Type":"ContainerStarted","Data":"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542"} Jul 20 13:28:09.120783 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.120679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" event={"ID":"0f822d67-92f4-4392-8fb4-b16b319902c6","Type":"ContainerStarted","Data":"48e87a27bc193fca644f4ce1f63a5e013f7275778a4bcb017f24d86a008a9be3"} Jul 20 13:28:09.120783 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.120733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:28:09.121378 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.121348 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:09.122332 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.122306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" event={"ID":"c8b6ba05-4af1-46f3-a522-f899de075c31","Type":"ContainerStarted","Data":"17db21529b3392731929afbb5c8f938e67039641be3a5366485bdf95c0b27b20"} Jul 20 13:28:09.122444 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.122349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" event={"ID":"c8b6ba05-4af1-46f3-a522-f899de075c31","Type":"ContainerStarted","Data":"82a7051694add52290e1d7d0fc9637fa009368f07d33a3016664479a63fdc430"} Jul 20 13:28:09.122444 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.122385 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:28:09.123923 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.123891 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:09.124224 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.124202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" event={"ID":"fea130e8-163c-45be-89fa-563f771976fc","Type":"ContainerStarted","Data":"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f"} Jul 20 13:28:09.124313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.124250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" event={"ID":"fea130e8-163c-45be-89fa-563f771976fc","Type":"ContainerStarted","Data":"c99e0fe694f50cbba0c5e052545287d282515e374b11f1bad89934ff95956146"} Jul 20 13:28:09.124313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.124294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:28:09.125659 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.125634 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:09.126329 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.126308 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:09.127161 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.127122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" event={"ID":"f18f9562-fe7e-422a-a77f-b48868ad8989","Type":"ContainerStarted","Data":"b26f82ed97593d41fdaa582b43b2c801f89407ac5d32302d5838c14c75149fa0"} Jul 20 13:28:09.128550 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.128527 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:09.129605 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.129585 2577 scope.go:117] "RemoveContainer" containerID="2518c929c981977a02cfad6beaf2652c56870eea648c7692e62da3d665783174" Jul 20 13:28:09.155676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.155633 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" podStartSLOduration=11.155618836 podStartE2EDuration="11.155618836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:09.152481685 +0000 UTC m=+688.216830599" watchObservedRunningTime="2026-07-20 13:28:09.155618836 +0000 UTC m=+688.219967801" Jul 20 13:28:09.185136 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.185061 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" podStartSLOduration=12.185050087 podStartE2EDuration="12.185050087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:09.184052979 +0000 UTC m=+688.248401905" watchObservedRunningTime="2026-07-20 13:28:09.185050087 +0000 UTC m=+688.249398980" Jul 20 13:28:09.186697 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.186676 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8gzxd\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8gzxd" Jul 20 13:28:09.209796 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.209752 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9scvc" podStartSLOduration=2.093636157 podStartE2EDuration="23.209737878s" totalImagesPullingTime="21.116101721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:09.207894409 +0000 UTC m=+688.272243323" watchObservedRunningTime="2026-07-20 13:28:09.209737878 +0000 UTC m=+688.274086834" Jul 20 13:28:09.211180 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:09.211144 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5xpfq\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5xpfq" Jul 20 13:28:09.241031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.240978 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" podStartSLOduration=12.240959768 podStartE2EDuration="12.240959768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:09.237308649 +0000 UTC m=+688.301657566" watchObservedRunningTime="2026-07-20 13:28:09.240959768 +0000 UTC m=+688.305308684" Jul 20 13:28:09.393060 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.393023 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b166c68-54d0-470c-93d7-a8ad5d7f1b04" path="/var/lib/kubelet/pods/5b166c68-54d0-470c-93d7-a8ad5d7f1b04/volumes" Jul 20 13:28:09.393451 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:09.393433 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabeff46-28a8-49ed-959c-ad276f0c2b4c" path="/var/lib/kubelet/pods/dabeff46-28a8-49ed-959c-ad276f0c2b4c/volumes" Jul 20 13:28:16.693631 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:16.693593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:20.136560 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.136526 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:28:20.137112 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.136613 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ccwxs" Jul 20 13:28:20.137112 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.136732 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:28:20.197389 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.197357 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:28:20.197634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.197601 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" podUID="fea130e8-163c-45be-89fa-563f771976fc" containerName="manager" containerID="cri-o://7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f" gracePeriod=10 Jul 20 13:28:20.462929 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.462904 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:28:20.525626 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.525599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5\") pod \"fea130e8-163c-45be-89fa-563f771976fc\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " Jul 20 13:28:20.525789 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.525662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume\") pod \"fea130e8-163c-45be-89fa-563f771976fc\" (UID: \"fea130e8-163c-45be-89fa-563f771976fc\") " Jul 20 13:28:20.526065 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.526039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fea130e8-163c-45be-89fa-563f771976fc" (UID: "fea130e8-163c-45be-89fa-563f771976fc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:28:20.527796 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.527770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5" (OuterVolumeSpecName: "kube-api-access-8r2h5") pod "fea130e8-163c-45be-89fa-563f771976fc" (UID: "fea130e8-163c-45be-89fa-563f771976fc"). InnerVolumeSpecName "kube-api-access-8r2h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:20.627158 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.627134 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fea130e8-163c-45be-89fa-563f771976fc-extensions-socket-volume\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:20.627158 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:20.627156 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2h5\" (UniqueName: \"kubernetes.io/projected/fea130e8-163c-45be-89fa-563f771976fc-kube-api-access-8r2h5\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:21.198740 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.198705 2577 generic.go:356] "Generic (PLEG): container finished" podID="fea130e8-163c-45be-89fa-563f771976fc" containerID="7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f" exitCode=0 Jul 20 13:28:21.199157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.198745 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" Jul 20 13:28:21.199157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.198787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" event={"ID":"fea130e8-163c-45be-89fa-563f771976fc","Type":"ContainerDied","Data":"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f"} Jul 20 13:28:21.199157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.198837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj" event={"ID":"fea130e8-163c-45be-89fa-563f771976fc","Type":"ContainerDied","Data":"c99e0fe694f50cbba0c5e052545287d282515e374b11f1bad89934ff95956146"} Jul 20 13:28:21.199157 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.198860 2577 scope.go:117] "RemoveContainer" containerID="7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f" Jul 20 13:28:21.208056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.208041 2577 scope.go:117] "RemoveContainer" containerID="7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f" Jul 20 13:28:21.208350 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:21.208328 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f\": container with ID starting with 7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f not found: ID does not exist" containerID="7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f" Jul 20 13:28:21.208454 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.208358 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f"} err="failed to get container status \"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f\": rpc error: code = NotFound desc = could not find container \"7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f\": container with ID starting with 7be078f76f504515bd5d4ea4ea55fd285da2928cf97de5b4da02d85fbd622b8f not found: ID does not exist" Jul 20 13:28:21.225650 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.225628 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:28:21.236348 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.236327 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-78mgj"] Jul 20 13:28:21.400408 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:21.400373 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fea130e8-163c-45be-89fa-563f771976fc" path="/var/lib/kubelet/pods/fea130e8-163c-45be-89fa-563f771976fc/volumes" Jul 20 13:28:27.845469 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:27.845417 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:27.911476 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:27.911446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:27.979003 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:27.978968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:36.643565 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.643484 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx"] Jul 20 13:28:36.644239 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.643989 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea130e8-163c-45be-89fa-563f771976fc" containerName="manager" Jul 20 13:28:36.644239 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.644010 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea130e8-163c-45be-89fa-563f771976fc" containerName="manager" Jul 20 13:28:36.644239 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.644143 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea130e8-163c-45be-89fa-563f771976fc" containerName="manager" Jul 20 13:28:36.650925 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.650904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.653726 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.653706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg7s5\"" Jul 20 13:28:36.653826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.653743 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 13:28:36.659703 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.659681 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx"] Jul 20 13:28:36.684580 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.684550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:36.772738 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.772738 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.772942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.772942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.772942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.772942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.773069 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.772974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.773069 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.773011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6jsq\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-kube-api-access-v6jsq\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.773069 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.773040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874003 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.873968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6jsq\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-kube-api-access-v6jsq\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874191 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874618 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874672 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874760 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.874864 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.874839 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.875330 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.875309 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.876803 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.876775 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.876803 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.876785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.882848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.882824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.883066 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.883045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6jsq\" (UniqueName: \"kubernetes.io/projected/086e3d2c-e7d0-40dc-95d7-de597c80f1a6-kube-api-access-v6jsq\") pod \"maas-default-gateway-openshift-default-56654dcf7-l4rvx\" (UID: \"086e3d2c-e7d0-40dc-95d7-de597c80f1a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:36.962938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:36.962839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:37.018832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:37.018800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:37.332486 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:37.332461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx"] Jul 20 13:28:37.334610 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:37.334575 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086e3d2c_e7d0_40dc_95d7_de597c80f1a6.slice/crio-2db5b54d2ddfa645b6ffe63e27d19c0af1168ad0726e66db15a13070dee285da WatchSource:0}: Error finding container 2db5b54d2ddfa645b6ffe63e27d19c0af1168ad0726e66db15a13070dee285da: Status 404 returned error can't find the container with id 2db5b54d2ddfa645b6ffe63e27d19c0af1168ad0726e66db15a13070dee285da Jul 20 13:28:38.303457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:38.303405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" event={"ID":"086e3d2c-e7d0-40dc-95d7-de597c80f1a6","Type":"ContainerStarted","Data":"2db5b54d2ddfa645b6ffe63e27d19c0af1168ad0726e66db15a13070dee285da"} Jul 20 13:28:40.192249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.192210 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:28:40.192543 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.192283 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:28:40.192543 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.192319 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:28:40.317509 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.317476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" event={"ID":"086e3d2c-e7d0-40dc-95d7-de597c80f1a6","Type":"ContainerStarted","Data":"4da3b90a2e979e5c5320fce0eae4d22aa8c85668d7735639aaeb92ce77bf1865"} Jul 20 13:28:40.340655 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.340589 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" podStartSLOduration=1.485223739 podStartE2EDuration="4.340567983s" totalImagesPullingTime="2.855344244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:40.338677347 +0000 UTC m=+719.403026263" watchObservedRunningTime="2026-07-20 13:28:40.340567983 +0000 UTC m=+719.404916907" Jul 20 13:28:40.963385 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:40.963349 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:41.411849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.411816 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:41.413578 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.413545 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:41.418841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.418810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.421402 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.421332 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 13:28:41.425533 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.425506 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:41.504825 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.504792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:41.511553 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.511522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:41.516585 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.516559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679jb\" (UniqueName: \"kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.516725 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.516592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.575400 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.575317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:41.617527 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.617487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679jb\" (UniqueName: \"kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.617733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.617540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.618466 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.618443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.626081 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.626057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-679jb\" (UniqueName: \"kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb\") pod \"limitador-limitador-64c94fd64f-fcw52\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.730701 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.730667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:41.874705 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.874673 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:41.878286 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:41.878250 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba3cf6f_3528_4e64_b7c5_f0eb076ffb50.slice/crio-28013036cfeb96c38b339c033730a65a14a95d31fb3145be63af91ceed6452a3 WatchSource:0}: Error finding container 28013036cfeb96c38b339c033730a65a14a95d31fb3145be63af91ceed6452a3: Status 404 returned error can't find the container with id 28013036cfeb96c38b339c033730a65a14a95d31fb3145be63af91ceed6452a3 Jul 20 13:28:41.968450 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:41.968412 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:42.230280 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:42.230199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:42.330129 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:42.330093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:42.332263 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:42.332238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" event={"ID":"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50","Type":"ContainerStarted","Data":"28013036cfeb96c38b339c033730a65a14a95d31fb3145be63af91ceed6452a3"} Jul 20 13:28:42.332602 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:42.332586 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:42.333560 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:42.333541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:28:45.351085 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:45.351046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" event={"ID":"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50","Type":"ContainerStarted","Data":"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0"} Jul 20 13:28:45.351516 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:45.351150 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:45.368676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:45.368618 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" podStartSLOduration=1.6379697439999998 podStartE2EDuration="4.368604412s" totalImagesPullingTime="2.730634668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:45.367233784 +0000 UTC m=+724.431582699" watchObservedRunningTime="2026-07-20 13:28:45.368604412 +0000 UTC m=+724.432953326" Jul 20 13:28:55.852720 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:55.852674 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:55.853264 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:55.852942 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" podUID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" containerName="limitador" containerID="cri-o://45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0" gracePeriod=30 Jul 20 13:28:55.853722 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:55.853701 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:56.411164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.411136 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:56.412000 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.411940 2577 generic.go:356] "Generic (PLEG): container finished" podID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" containerID="45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0" exitCode=0 Jul 20 13:28:56.412121 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.412011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" event={"ID":"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50","Type":"ContainerDied","Data":"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0"} Jul 20 13:28:56.412121 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.412088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" event={"ID":"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50","Type":"ContainerDied","Data":"28013036cfeb96c38b339c033730a65a14a95d31fb3145be63af91ceed6452a3"} Jul 20 13:28:56.412260 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.412122 2577 scope.go:117] "RemoveContainer" containerID="45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0" Jul 20 13:28:56.420718 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.420698 2577 scope.go:117] "RemoveContainer" containerID="45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0" Jul 20 13:28:56.420986 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:28:56.420959 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0\": container with ID starting with 45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0 not found: ID does not exist" containerID="45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0" Jul 20 13:28:56.421067 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.420985 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0"} err="failed to get container status \"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0\": rpc error: code = NotFound desc = could not find container \"45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0\": container with ID starting with 45a99d73cf3f4bdda321ae16b20247d620da062126f8bdc7abf8860cdf43cba0 not found: ID does not exist" Jul 20 13:28:56.563214 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.563157 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file\") pod \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " Jul 20 13:28:56.563370 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.563242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-679jb\" (UniqueName: \"kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb\") pod \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\" (UID: \"cba3cf6f-3528-4e64-b7c5-f0eb076ffb50\") " Jul 20 13:28:56.563543 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.563507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file" (OuterVolumeSpecName: "config-file") pod "cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" (UID: "cba3cf6f-3528-4e64-b7c5-f0eb076ffb50"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:28:56.563680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.563579 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-config-file\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:56.565405 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.565384 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb" (OuterVolumeSpecName: "kube-api-access-679jb") pod "cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" (UID: "cba3cf6f-3528-4e64-b7c5-f0eb076ffb50"). InnerVolumeSpecName "kube-api-access-679jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:56.664894 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:56.664806 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-679jb\" (UniqueName: \"kubernetes.io/projected/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50-kube-api-access-679jb\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:28:57.418083 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:57.418058 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fcw52" Jul 20 13:28:57.435997 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:57.435963 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:57.439990 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:57.439959 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fcw52"] Jul 20 13:28:57.564837 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:57.564802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:57.656704 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:57.656618 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:58.095733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.095640 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l"] Jul 20 13:28:58.096038 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.096025 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" containerName="limitador" Jul 20 13:28:58.096109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.096039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" containerName="limitador" Jul 20 13:28:58.096109 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.096098 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" containerName="limitador" Jul 20 13:28:58.100743 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.100721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.103557 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.103531 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 13:28:58.103974 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.103952 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 20 13:28:58.104230 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.104205 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9l9gp\"" Jul 20 13:28:58.104567 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.104549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 13:28:58.107335 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.107307 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l"] Jul 20 13:28:58.125495 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.125462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:28:58.279742 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.279704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmm9\" (UniqueName: \"kubernetes.io/projected/d5dd00ef-7212-4679-a886-24b79cbee889-kube-api-access-shmm9\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.279917 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.279844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd00ef-7212-4679-a886-24b79cbee889-data\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.380903 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.380861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd00ef-7212-4679-a886-24b79cbee889-data\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.381091 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.380919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmm9\" (UniqueName: \"kubernetes.io/projected/d5dd00ef-7212-4679-a886-24b79cbee889-kube-api-access-shmm9\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.381425 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.381407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dd00ef-7212-4679-a886-24b79cbee889-data\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.389483 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.389457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmm9\" (UniqueName: \"kubernetes.io/projected/d5dd00ef-7212-4679-a886-24b79cbee889-kube-api-access-shmm9\") pod \"postgres-69fb4594cf-qmz4l\" (UID: \"d5dd00ef-7212-4679-a886-24b79cbee889\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.413287 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.413253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:28:58.549096 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:58.549062 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l"] Jul 20 13:28:58.551147 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:28:58.551122 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dd00ef_7212_4679_a886_24b79cbee889.slice/crio-8ccb5f1b90698a8742ba9abf912b3362d3467f2b4cd7dc6dc6a44f5aa2f222bd WatchSource:0}: Error finding container 8ccb5f1b90698a8742ba9abf912b3362d3467f2b4cd7dc6dc6a44f5aa2f222bd: Status 404 returned error can't find the container with id 8ccb5f1b90698a8742ba9abf912b3362d3467f2b4cd7dc6dc6a44f5aa2f222bd Jul 20 13:28:59.397200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:59.396919 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba3cf6f-3528-4e64-b7c5-f0eb076ffb50" path="/var/lib/kubelet/pods/cba3cf6f-3528-4e64-b7c5-f0eb076ffb50/volumes" Jul 20 13:28:59.434926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:28:59.434848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" event={"ID":"d5dd00ef-7212-4679-a886-24b79cbee889","Type":"ContainerStarted","Data":"8ccb5f1b90698a8742ba9abf912b3362d3467f2b4cd7dc6dc6a44f5aa2f222bd"} Jul 20 13:29:04.466680 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:04.466631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" event={"ID":"d5dd00ef-7212-4679-a886-24b79cbee889","Type":"ContainerStarted","Data":"45087f684174f80ce4c6736a3a55062567f595392eeab9b6e1f625852c177cb1"} Jul 20 13:29:04.467074 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:04.466755 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:29:04.488412 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:04.488353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" podStartSLOduration=1.288708361 podStartE2EDuration="6.488332922s" totalImagesPullingTime="5.199624561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:29:04.484951755 +0000 UTC m=+743.549300676" watchObservedRunningTime="2026-07-20 13:29:04.488332922 +0000 UTC m=+743.552681838" Jul 20 13:29:10.502834 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:10.502803 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qmz4l" Jul 20 13:29:10.872233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:10.872203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:11.242156 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.242060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:11.378644 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.378607 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d5cc8fb4c-8j225"] Jul 20 13:29:11.382508 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.382485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.384908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.384889 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-94vrh\"" Jul 20 13:29:11.387741 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.387713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gmm\" (UniqueName: \"kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm\") pod \"authorino-d5cc8fb4c-8j225\" (UID: \"f7560ba3-9948-434e-8e77-eaf6f9cc8525\") " pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.395663 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.395637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5cc8fb4c-8j225"] Jul 20 13:29:11.451696 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.451657 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5cc8fb4c-8j225"] Jul 20 13:29:11.451986 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:11.451961 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-44gmm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-d5cc8fb4c-8j225" podUID="f7560ba3-9948-434e-8e77-eaf6f9cc8525" Jul 20 13:29:11.484357 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.484328 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:29:11.489431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.489390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gmm\" (UniqueName: \"kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm\") pod \"authorino-d5cc8fb4c-8j225\" (UID: \"f7560ba3-9948-434e-8e77-eaf6f9cc8525\") " pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.491530 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.491502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.495233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.495131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 13:29:11.497894 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.497871 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:29:11.499690 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.499656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gmm\" (UniqueName: \"kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm\") pod \"authorino-d5cc8fb4c-8j225\" (UID: \"f7560ba3-9948-434e-8e77-eaf6f9cc8525\") " pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.507449 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.507426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.519164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.519144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:11.691464 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.691430 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44gmm\" (UniqueName: \"kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm\") pod \"f7560ba3-9948-434e-8e77-eaf6f9cc8525\" (UID: \"f7560ba3-9948-434e-8e77-eaf6f9cc8525\") " Jul 20 13:29:11.691642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.691579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.691642 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.691630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5t2\" (UniqueName: \"kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.693556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.693534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm" (OuterVolumeSpecName: "kube-api-access-44gmm") pod "f7560ba3-9948-434e-8e77-eaf6f9cc8525" (UID: "f7560ba3-9948-434e-8e77-eaf6f9cc8525"). InnerVolumeSpecName "kube-api-access-44gmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:29:11.792463 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.792382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.792463 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.792451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5t2\" (UniqueName: \"kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.792643 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.792497 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gmm\" (UniqueName: \"kubernetes.io/projected/f7560ba3-9948-434e-8e77-eaf6f9cc8525-kube-api-access-44gmm\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:29:11.794969 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.794946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.800783 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.800760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5t2\" (UniqueName: \"kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2\") pod \"authorino-646cfcbfff-dhsvm\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.816585 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.816562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:29:11.953262 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:11.953235 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:29:11.955945 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:29:11.955917 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9d35ce2_ada9_4ac7_b066_aea33bb25d1c.slice/crio-4df613f2d5488aef28fce7b73dc0f7ccb74577e7ebe26d07c03bfd89b367aebe WatchSource:0}: Error finding container 4df613f2d5488aef28fce7b73dc0f7ccb74577e7ebe26d07c03bfd89b367aebe: Status 404 returned error can't find the container with id 4df613f2d5488aef28fce7b73dc0f7ccb74577e7ebe26d07c03bfd89b367aebe Jul 20 13:29:12.514561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:12.514538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5cc8fb4c-8j225" Jul 20 13:29:12.514926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:12.514555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" event={"ID":"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c","Type":"ContainerStarted","Data":"4df613f2d5488aef28fce7b73dc0f7ccb74577e7ebe26d07c03bfd89b367aebe"} Jul 20 13:29:12.545459 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:12.545428 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5cc8fb4c-8j225"] Jul 20 13:29:12.548545 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:12.548516 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d5cc8fb4c-8j225"] Jul 20 13:29:13.394343 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:13.394304 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7560ba3-9948-434e-8e77-eaf6f9cc8525" path="/var/lib/kubelet/pods/f7560ba3-9948-434e-8e77-eaf6f9cc8525/volumes" Jul 20 13:29:16.542388 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:16.542293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" event={"ID":"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c","Type":"ContainerStarted","Data":"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b"} Jul 20 13:29:21.487652 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:21.487606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:21.550303 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:21.550270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:41.557816 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.557749 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" podStartSLOduration=26.271168971 podStartE2EDuration="30.557727649s" totalImagesPullingTime="4.286558678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:29:16.556765706 +0000 UTC m=+755.621114622" watchObservedRunningTime="2026-07-20 13:29:41.557727649 +0000 UTC m=+780.622076565" Jul 20 13:29:41.558497 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.558478 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:29:41.563119 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.563101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.566199 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:41.566157 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" Jul 20 13:29:41.570585 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:41.570563 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-dockercfg-mhvsf\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvsf\"" type="*v1.Secret" Jul 20 13:29:41.570686 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:41.570598 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"maas-parameters\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" Jul 20 13:29:41.570789 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:41.570771 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-controller-6f4b8bc958-lsl6f\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.604098 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.604046 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:29:41.622262 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.622234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:41.641221 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.641198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46bq\" (UniqueName: \"kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.641298 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.641263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.741835 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.741808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.741994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.741859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46bq\" (UniqueName: \"kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.752848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.752821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46bq\" (UniqueName: \"kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:41.758622 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:41.758598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:42.742306 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:42.742278 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: failed to sync secret cache: timed out waiting for the condition Jul 20 13:29:42.742686 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:29:42.742352 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert podName:f21473d0-6821-4e5b-b8b9-8f529e3cf91a nodeName:}" failed. No retries permitted until 2026-07-20 13:29:43.242337375 +0000 UTC m=+782.306686268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert") pod "maas-controller-6f4b8bc958-lsl6f" (UID: "f21473d0-6821-4e5b-b8b9-8f529e3cf91a") : failed to sync secret cache: timed out waiting for the condition Jul 20 13:29:42.767940 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:42.767914 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvsf\"" Jul 20 13:29:43.009296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.009219 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 13:29:43.012409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.012388 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 13:29:43.254750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.254711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:43.257120 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.257092 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") pod \"maas-controller-6f4b8bc958-lsl6f\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:43.372846 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.372825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:43.512814 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.512784 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:29:43.513419 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:29:43.513390 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21473d0_6821_4e5b_b8b9_8f529e3cf91a.slice/crio-0246385da9a9efa56c0a5267c333a87a6c3324347fa529ac5d261fde6b62c019 WatchSource:0}: Error finding container 0246385da9a9efa56c0a5267c333a87a6c3324347fa529ac5d261fde6b62c019: Status 404 returned error can't find the container with id 0246385da9a9efa56c0a5267c333a87a6c3324347fa529ac5d261fde6b62c019 Jul 20 13:29:43.693516 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:43.693434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" event={"ID":"f21473d0-6821-4e5b-b8b9-8f529e3cf91a","Type":"ContainerStarted","Data":"0246385da9a9efa56c0a5267c333a87a6c3324347fa529ac5d261fde6b62c019"} Jul 20 13:29:46.525136 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.525103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:46.618113 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.618058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:29:46.713898 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.713865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" event={"ID":"f21473d0-6821-4e5b-b8b9-8f529e3cf91a","Type":"ContainerStarted","Data":"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719"} Jul 20 13:29:46.714063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.714007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:46.721215 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.721188 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:29:46.725458 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.725438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.737754 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.737729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:29:46.747394 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.747353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" podStartSLOduration=3.480419431 podStartE2EDuration="5.747340012s" totalImagesPullingTime="2.266920581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:29:46.745135284 +0000 UTC m=+785.809484201" watchObservedRunningTime="2026-07-20 13:29:46.747340012 +0000 UTC m=+785.811688927" Jul 20 13:29:46.788930 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.788856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9nx\" (UniqueName: \"kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.789055 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.788972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.890362 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.890325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9nx\" (UniqueName: \"kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.890532 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.890374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.892918 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.892895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:46.899649 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:46.899624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9nx\" (UniqueName: \"kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx\") pod \"maas-controller-6f55c6d57f-77ntv\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:47.037226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:47.037185 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:47.183421 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:47.183394 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:29:47.185238 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:29:47.185209 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366c820_9c47_4d7d_ba14_900dd5cc3c35.slice/crio-cb5f02b89f1ae388109fc9b6e26b547b6459c3ada1095ace8ed7db1eeb25d0cb WatchSource:0}: Error finding container cb5f02b89f1ae388109fc9b6e26b547b6459c3ada1095ace8ed7db1eeb25d0cb: Status 404 returned error can't find the container with id cb5f02b89f1ae388109fc9b6e26b547b6459c3ada1095ace8ed7db1eeb25d0cb Jul 20 13:29:47.721409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:47.721369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" event={"ID":"7366c820-9c47-4d7d-ba14-900dd5cc3c35","Type":"ContainerStarted","Data":"cb5f02b89f1ae388109fc9b6e26b547b6459c3ada1095ace8ed7db1eeb25d0cb"} Jul 20 13:29:48.728148 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:48.728114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" event={"ID":"7366c820-9c47-4d7d-ba14-900dd5cc3c35","Type":"ContainerStarted","Data":"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d"} Jul 20 13:29:48.728565 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:48.728226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:48.751340 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:48.751294 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" podStartSLOduration=2.214088229 podStartE2EDuration="2.751280042s" totalImagesPullingTime="537.191813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:29:48.749413776 +0000 UTC m=+787.813762692" watchObservedRunningTime="2026-07-20 13:29:48.751280042 +0000 UTC m=+787.815628968" Jul 20 13:29:57.724318 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:57.724290 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:29:59.736526 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:59.736497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:29:59.782162 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:59.782130 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:29:59.782368 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:29:59.782322 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" containerName="manager" containerID="cri-o://f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719" gracePeriod=30 Jul 20 13:30:00.031565 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.031540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:30:00.206889 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.206852 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") pod \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " Jul 20 13:30:00.207056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.206935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r46bq\" (UniqueName: \"kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq\") pod \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\" (UID: \"f21473d0-6821-4e5b-b8b9-8f529e3cf91a\") " Jul 20 13:30:00.209017 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.208985 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f21473d0-6821-4e5b-b8b9-8f529e3cf91a" (UID: "f21473d0-6821-4e5b-b8b9-8f529e3cf91a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:30:00.209129 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.209019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq" (OuterVolumeSpecName: "kube-api-access-r46bq") pod "f21473d0-6821-4e5b-b8b9-8f529e3cf91a" (UID: "f21473d0-6821-4e5b-b8b9-8f529e3cf91a"). InnerVolumeSpecName "kube-api-access-r46bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:30:00.308442 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.308368 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46bq\" (UniqueName: \"kubernetes.io/projected/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-kube-api-access-r46bq\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:30:00.308442 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.308396 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21473d0-6821-4e5b-b8b9-8f529e3cf91a-webhook-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:30:00.797354 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.797323 2577 generic.go:356] "Generic (PLEG): container finished" podID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" containerID="f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719" exitCode=0 Jul 20 13:30:00.797833 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.797364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" event={"ID":"f21473d0-6821-4e5b-b8b9-8f529e3cf91a","Type":"ContainerDied","Data":"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719"} Jul 20 13:30:00.797833 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.797374 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" Jul 20 13:30:00.797833 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.797408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4b8bc958-lsl6f" event={"ID":"f21473d0-6821-4e5b-b8b9-8f529e3cf91a","Type":"ContainerDied","Data":"0246385da9a9efa56c0a5267c333a87a6c3324347fa529ac5d261fde6b62c019"} Jul 20 13:30:00.797833 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.797429 2577 scope.go:117] "RemoveContainer" containerID="f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719" Jul 20 13:30:00.806816 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.806798 2577 scope.go:117] "RemoveContainer" containerID="f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719" Jul 20 13:30:00.807061 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:30:00.807044 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719\": container with ID starting with f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719 not found: ID does not exist" containerID="f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719" Jul 20 13:30:00.807101 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.807070 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719"} err="failed to get container status \"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719\": rpc error: code = NotFound desc = could not find container \"f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719\": container with ID starting with f4c13ac573a2f034f9ade5e9248394b4aa620126cdb3b0ca744ed1693abb0719 not found: ID does not exist" Jul 20 13:30:00.818985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.818959 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:30:00.822604 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:00.822586 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f4b8bc958-lsl6f"] Jul 20 13:30:01.393899 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:01.393867 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" path="/var/lib/kubelet/pods/f21473d0-6821-4e5b-b8b9-8f529e3cf91a/volumes" Jul 20 13:30:20.710740 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:20.710658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:20.868738 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:20.868701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:21.356290 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.356259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:21.558545 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.558511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:21.748068 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.748032 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:21.849470 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.849434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:21.861606 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.861569 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pkj5z"] Jul 20 13:30:21.862094 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.862074 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" containerName="manager" Jul 20 13:30:21.862094 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.862096 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" containerName="manager" Jul 20 13:30:21.862297 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.862221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21473d0-6821-4e5b-b8b9-8f529e3cf91a" containerName="manager" Jul 20 13:30:21.870634 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.870609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:21.873305 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.873285 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sh8z4"] Jul 20 13:30:21.875463 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.875446 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 20 13:30:21.875555 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.875450 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g42hd\"" Jul 20 13:30:21.877854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.877836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:21.881729 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.881702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pkj5z"] Jul 20 13:30:21.892152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.892063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sh8z4"] Jul 20 13:30:21.981962 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.981928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w655l\" (UniqueName: \"kubernetes.io/projected/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-kube-api-access-w655l\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:21.982131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.982066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:21.982131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.982110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9rr\" (UniqueName: \"kubernetes.io/projected/489ec3f6-112d-47fd-b801-2cf3667c1a9f-kube-api-access-pt9rr\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:21.982264 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:21.982144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489ec3f6-112d-47fd-b801-2cf3667c1a9f-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.083292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.083204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w655l\" (UniqueName: \"kubernetes.io/projected/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-kube-api-access-w655l\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:22.083292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.083290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:22.083535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.083314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9rr\" (UniqueName: \"kubernetes.io/projected/489ec3f6-112d-47fd-b801-2cf3667c1a9f-kube-api-access-pt9rr\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.083535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.083343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489ec3f6-112d-47fd-b801-2cf3667c1a9f-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.084012 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.083988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489ec3f6-112d-47fd-b801-2cf3667c1a9f-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.084108 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.084020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:22.092617 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.092586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9rr\" (UniqueName: \"kubernetes.io/projected/489ec3f6-112d-47fd-b801-2cf3667c1a9f-kube-api-access-pt9rr\") pod \"payload-processing-5bf47cd988-sh8z4\" (UID: \"489ec3f6-112d-47fd-b801-2cf3667c1a9f\") " pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.092720 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.092621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w655l\" (UniqueName: \"kubernetes.io/projected/f6fdefa1-5265-4ac0-82a0-53db3c1d8955-kube-api-access-w655l\") pod \"payload-pre-processing-7677b774bb-pkj5z\" (UID: \"f6fdefa1-5265-4ac0-82a0-53db3c1d8955\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:22.183953 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.183912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:22.193638 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.193616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:22.376587 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.376562 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sh8z4"] Jul 20 13:30:22.378494 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:30:22.378461 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489ec3f6_112d_47fd_b801_2cf3667c1a9f.slice/crio-73a5f4ae49b061cf5cccf9d950985bc9ac68063f72ba38640bef9ffb6c9c3955 WatchSource:0}: Error finding container 73a5f4ae49b061cf5cccf9d950985bc9ac68063f72ba38640bef9ffb6c9c3955: Status 404 returned error can't find the container with id 73a5f4ae49b061cf5cccf9d950985bc9ac68063f72ba38640bef9ffb6c9c3955 Jul 20 13:30:22.550015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.549990 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pkj5z"] Jul 20 13:30:22.551469 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:30:22.551444 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fdefa1_5265_4ac0_82a0_53db3c1d8955.slice/crio-1f7e0d91ee24f42f7c382d6a5e9130af9577976b7e6c38d103b9ca0b6930fdf8 WatchSource:0}: Error finding container 1f7e0d91ee24f42f7c382d6a5e9130af9577976b7e6c38d103b9ca0b6930fdf8: Status 404 returned error can't find the container with id 1f7e0d91ee24f42f7c382d6a5e9130af9577976b7e6c38d103b9ca0b6930fdf8 Jul 20 13:30:22.927789 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.927752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" event={"ID":"489ec3f6-112d-47fd-b801-2cf3667c1a9f","Type":"ContainerStarted","Data":"73a5f4ae49b061cf5cccf9d950985bc9ac68063f72ba38640bef9ffb6c9c3955"} Jul 20 13:30:22.929223 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:22.929193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" event={"ID":"f6fdefa1-5265-4ac0-82a0-53db3c1d8955","Type":"ContainerStarted","Data":"1f7e0d91ee24f42f7c382d6a5e9130af9577976b7e6c38d103b9ca0b6930fdf8"} Jul 20 13:30:24.945705 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.945660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" event={"ID":"489ec3f6-112d-47fd-b801-2cf3667c1a9f","Type":"ContainerStarted","Data":"9bc740b5faf70d5fb0cc7f6e9af207eb32cd06213885d90f5baa7ba295109f7a"} Jul 20 13:30:24.946202 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.945772 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:24.947511 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.947484 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" event={"ID":"f6fdefa1-5265-4ac0-82a0-53db3c1d8955","Type":"ContainerStarted","Data":"be88de332677f0a9dad116d835328cd5763c33390204814e959f9e6380743491"} Jul 20 13:30:24.947639 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.947572 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:24.964872 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.964817 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" podStartSLOduration=1.578115894 podStartE2EDuration="3.964801119s" totalImagesPullingTime="2.386685225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:24.962929405 +0000 UTC m=+824.027278321" watchObservedRunningTime="2026-07-20 13:30:24.964801119 +0000 UTC m=+824.029150034" Jul 20 13:30:24.980281 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:24.980220 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" podStartSLOduration=1.760758649 podStartE2EDuration="3.980206188s" totalImagesPullingTime="2.219447539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:24.979422066 +0000 UTC m=+824.043770981" watchObservedRunningTime="2026-07-20 13:30:24.980206188 +0000 UTC m=+824.044555102" Jul 20 13:30:28.374307 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:28.374272 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l4rvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 13:30:28.374789 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:28.374343 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" podUID="086e3d2c-e7d0-40dc-95d7-de597c80f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 13:30:28.623182 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:28.623121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:29.788409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.788318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:29.889917 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.889885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:29.978771 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.978743 2577 generic.go:356] "Generic (PLEG): container finished" podID="086e3d2c-e7d0-40dc-95d7-de597c80f1a6" containerID="4da3b90a2e979e5c5320fce0eae4d22aa8c85668d7735639aaeb92ce77bf1865" exitCode=0 Jul 20 13:30:29.978921 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.978799 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" event={"ID":"086e3d2c-e7d0-40dc-95d7-de597c80f1a6","Type":"ContainerDied","Data":"4da3b90a2e979e5c5320fce0eae4d22aa8c85668d7735639aaeb92ce77bf1865"} Jul 20 13:30:29.979165 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.979151 2577 scope.go:117] "RemoveContainer" containerID="4da3b90a2e979e5c5320fce0eae4d22aa8c85668d7735639aaeb92ce77bf1865" Jul 20 13:30:29.980154 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.980120 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:30:29.980233 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.980215 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:30:29.980269 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:29.980249 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:30:30.986929 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:30.986895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" event={"ID":"086e3d2c-e7d0-40dc-95d7-de597c80f1a6","Type":"ContainerStarted","Data":"3eef818f253a50b2d6862411adee77d11d63683f2fb38efef6157ed2cd5286d1"} Jul 20 13:30:31.963245 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:31.963206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:30:31.963465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:31.963438 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l4rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 20 13:30:31.963552 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:31.963501 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" podUID="086e3d2c-e7d0-40dc-95d7-de597c80f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 20 13:30:32.177495 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:32.177461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:32.269200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:32.269101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:32.963798 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:32.963770 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l4rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 20 13:30:32.963941 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:32.963826 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" podUID="086e3d2c-e7d0-40dc-95d7-de597c80f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 20 13:30:33.964249 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:33.964210 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l4rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 20 13:30:33.964638 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:33.964274 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" podUID="086e3d2c-e7d0-40dc-95d7-de597c80f1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 20 13:30:34.967131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:34.967097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:30:34.967609 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:34.967389 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:30:34.968092 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:34.968074 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l4rvx" Jul 20 13:30:35.954951 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:35.954923 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-sh8z4" Jul 20 13:30:35.955316 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:35.955294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pkj5z" Jul 20 13:30:40.186669 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:40.186631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:51.754876 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:51.754847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:30:56.297938 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:30:56.297899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:31:07.947358 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.947328 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7"] Jul 20 13:31:07.950908 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.950891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:07.954278 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.954254 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 13:31:07.954409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.954281 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 20 13:31:07.954409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.954257 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 13:31:07.954409 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.954373 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f779n\"" Jul 20 13:31:07.961455 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.961434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7"] Jul 20 13:31:07.971158 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:07.971135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:31:08.016579 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.016716 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.016716 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.016801 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.016801 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116c30bf-454e-4905-ac6b-850b8666b348-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.016801 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.016785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fkh\" (UniqueName: \"kubernetes.io/projected/116c30bf-454e-4905-ac6b-850b8666b348-kube-api-access-f8fkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118060 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fkh\" (UniqueName: \"kubernetes.io/projected/116c30bf-454e-4905-ac6b-850b8666b348-kube-api-access-f8fkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118285 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118285 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118285 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118285 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118508 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116c30bf-454e-4905-ac6b-850b8666b348-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118598 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.118665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.118644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.121295 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.121269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116c30bf-454e-4905-ac6b-850b8666b348-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.121445 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.121387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116c30bf-454e-4905-ac6b-850b8666b348-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.126028 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.126007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fkh\" (UniqueName: \"kubernetes.io/projected/116c30bf-454e-4905-ac6b-850b8666b348-kube-api-access-f8fkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7\" (UID: \"116c30bf-454e-4905-ac6b-850b8666b348\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.262289 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.262218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:08.400509 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:08.400475 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7"] Jul 20 13:31:08.401666 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:31:08.401635 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116c30bf_454e_4905_ac6b_850b8666b348.slice/crio-a4f939b355af93edb2629301d6030be9d62ec02f3c8d5fceca49886b2fa829b8 WatchSource:0}: Error finding container a4f939b355af93edb2629301d6030be9d62ec02f3c8d5fceca49886b2fa829b8: Status 404 returned error can't find the container with id a4f939b355af93edb2629301d6030be9d62ec02f3c8d5fceca49886b2fa829b8 Jul 20 13:31:09.214226 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:09.214189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" event={"ID":"116c30bf-454e-4905-ac6b-850b8666b348","Type":"ContainerStarted","Data":"a4f939b355af93edb2629301d6030be9d62ec02f3c8d5fceca49886b2fa829b8"} Jul 20 13:31:10.224379 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:10.224338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" event={"ID":"116c30bf-454e-4905-ac6b-850b8666b348","Type":"ContainerStarted","Data":"a3769427eedec9658e04f1e5cb4446ca61cdbd744e0bffc70117c899b259d252"} Jul 20 13:31:10.247771 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:10.247720 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" podStartSLOduration=2.200564466 podStartE2EDuration="3.247705014s" totalImagesPullingTime="1.047140548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:31:10.245523164 +0000 UTC m=+869.309872080" watchObservedRunningTime="2026-07-20 13:31:10.247705014 +0000 UTC m=+869.312053928" Jul 20 13:31:16.848722 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.848688 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf"] Jul 20 13:31:16.853279 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.853257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.855918 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.855890 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 20 13:31:16.860637 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.860615 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf"] Jul 20 13:31:16.880493 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.880464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:31:16.895204 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.895339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.895413 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.895413 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9s7\" (UniqueName: \"kubernetes.io/projected/b88d16f3-9f28-4108-8eeb-c324fe48fae2-kube-api-access-7h9s7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.895525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.895525 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.895499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.996950 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.996911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.996966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9s7\" (UniqueName: \"kubernetes.io/projected/b88d16f3-9f28-4108-8eeb-c324fe48fae2-kube-api-access-7h9s7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997471 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997471 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.997644 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.997624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.999523 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.999500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88d16f3-9f28-4108-8eeb-c324fe48fae2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:16.999762 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:16.999746 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88d16f3-9f28-4108-8eeb-c324fe48fae2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:17.005676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:17.005654 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9s7\" (UniqueName: \"kubernetes.io/projected/b88d16f3-9f28-4108-8eeb-c324fe48fae2-kube-api-access-7h9s7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnsvf\" (UID: \"b88d16f3-9f28-4108-8eeb-c324fe48fae2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:17.165582 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:17.165548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:17.312108 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:17.312084 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf"] Jul 20 13:31:17.312926 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:31:17.312903 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88d16f3_9f28_4108_8eeb_c324fe48fae2.slice/crio-2799887b5b04217c7b499864d1c9af528e9498801657b2e33e7b7ce48dced111 WatchSource:0}: Error finding container 2799887b5b04217c7b499864d1c9af528e9498801657b2e33e7b7ce48dced111: Status 404 returned error can't find the container with id 2799887b5b04217c7b499864d1c9af528e9498801657b2e33e7b7ce48dced111 Jul 20 13:31:18.262846 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.262814 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:18.263360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.262911 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:18.276131 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.276105 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:18.276864 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.276831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" event={"ID":"b88d16f3-9f28-4108-8eeb-c324fe48fae2","Type":"ContainerStarted","Data":"dc83941ad9d5f69f05ad93e1e819b48d89419fa9b80b64f3869bdb4d781a109e"} Jul 20 13:31:18.277008 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.276888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" event={"ID":"b88d16f3-9f28-4108-8eeb-c324fe48fae2","Type":"ContainerStarted","Data":"2799887b5b04217c7b499864d1c9af528e9498801657b2e33e7b7ce48dced111"} Jul 20 13:31:18.288410 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.288381 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7" Jul 20 13:31:18.321089 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:18.321037 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" podStartSLOduration=2.146259493 podStartE2EDuration="2.321020386s" totalImagesPullingTime="174.760893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:31:18.318871095 +0000 UTC m=+877.383220013" watchObservedRunningTime="2026-07-20 13:31:18.321020386 +0000 UTC m=+877.385369342" Jul 20 13:31:21.877862 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:21.877824 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:31:27.166570 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:27.166515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:27.167216 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:27.166592 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:27.182732 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:27.182703 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:27.345942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:27.345912 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnsvf" Jul 20 13:31:47.183322 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:47.183296 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:31:47.186014 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:47.185981 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:31:47.190921 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:47.190897 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:31:47.193383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:47.193363 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:31:50.129121 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:50.129089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:31:50.184225 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:31:50.184192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:09.787981 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:09.787943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:13.147819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:13.147784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:45.031210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:45.031152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:45.123581 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:45.123549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:45.981000 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:45.980959 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:32:45.981204 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:45.981154 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" podUID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" containerName="authorino" containerID="cri-o://c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b" gracePeriod=30 Jul 20 13:32:46.238550 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.238479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:46.244880 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.244860 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:32:46.321389 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.321362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz5t2\" (UniqueName: \"kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2\") pod \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " Jul 20 13:32:46.321556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.321472 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert\") pod \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\" (UID: \"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c\") " Jul 20 13:32:46.323600 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.323571 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2" (OuterVolumeSpecName: "kube-api-access-rz5t2") pod "e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" (UID: "e9d35ce2-ada9-4ac7-b066-aea33bb25d1c"). InnerVolumeSpecName "kube-api-access-rz5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:32:46.334423 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.334392 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" (UID: "e9d35ce2-ada9-4ac7-b066-aea33bb25d1c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:32:46.422339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.422307 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-tls-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:32:46.422339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.422333 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5t2\" (UniqueName: \"kubernetes.io/projected/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c-kube-api-access-rz5t2\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:32:46.425815 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.425793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:32:46.786259 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.786226 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" containerID="c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b" exitCode=0 Jul 20 13:32:46.786431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.786270 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" Jul 20 13:32:46.786431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.786283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" event={"ID":"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c","Type":"ContainerDied","Data":"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b"} Jul 20 13:32:46.786431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.786314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-646cfcbfff-dhsvm" event={"ID":"e9d35ce2-ada9-4ac7-b066-aea33bb25d1c","Type":"ContainerDied","Data":"4df613f2d5488aef28fce7b73dc0f7ccb74577e7ebe26d07c03bfd89b367aebe"} Jul 20 13:32:46.786431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.786330 2577 scope.go:117] "RemoveContainer" containerID="c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b" Jul 20 13:32:46.797221 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.797189 2577 scope.go:117] "RemoveContainer" containerID="c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b" Jul 20 13:32:46.797561 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:32:46.797534 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b\": container with ID starting with c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b not found: ID does not exist" containerID="c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b" Jul 20 13:32:46.797682 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.797570 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b"} err="failed to get container status \"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b\": rpc error: code = NotFound desc = could not find container \"c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b\": container with ID starting with c01c0a703fbc9f4055f81293c3077ca3ac39a65fc20622b71a2d7649241ba81b not found: ID does not exist" Jul 20 13:32:46.815649 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.815597 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:32:46.822137 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:46.822110 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-646cfcbfff-dhsvm"] Jul 20 13:32:47.394423 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:32:47.394387 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" path="/var/lib/kubelet/pods/e9d35ce2-ada9-4ac7-b066-aea33bb25d1c/volumes" Jul 20 13:33:01.627034 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:01.627000 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:02.397085 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:02.397050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:13.167944 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:13.167908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:13.850344 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:13.850312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:16.470621 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:16.470581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:16.534594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:16.534561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:33:47.242431 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.242398 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:33:47.242878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.242649 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" podUID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" containerName="manager" containerID="cri-o://49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d" gracePeriod=30 Jul 20 13:33:47.505599 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.505539 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:33:47.580545 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.580516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w9nx\" (UniqueName: \"kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx\") pod \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " Jul 20 13:33:47.580723 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.580624 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert\") pod \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\" (UID: \"7366c820-9c47-4d7d-ba14-900dd5cc3c35\") " Jul 20 13:33:47.582471 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.582438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx" (OuterVolumeSpecName: "kube-api-access-4w9nx") pod "7366c820-9c47-4d7d-ba14-900dd5cc3c35" (UID: "7366c820-9c47-4d7d-ba14-900dd5cc3c35"). InnerVolumeSpecName "kube-api-access-4w9nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:33:47.582589 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.582470 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7366c820-9c47-4d7d-ba14-900dd5cc3c35" (UID: "7366c820-9c47-4d7d-ba14-900dd5cc3c35"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:33:47.682242 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.682199 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9nx\" (UniqueName: \"kubernetes.io/projected/7366c820-9c47-4d7d-ba14-900dd5cc3c35-kube-api-access-4w9nx\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:33:47.682242 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:47.682238 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7366c820-9c47-4d7d-ba14-900dd5cc3c35-webhook-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:33:48.148258 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.148221 2577 generic.go:356] "Generic (PLEG): container finished" podID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" containerID="49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d" exitCode=0 Jul 20 13:33:48.148426 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.148263 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" Jul 20 13:33:48.148426 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.148303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" event={"ID":"7366c820-9c47-4d7d-ba14-900dd5cc3c35","Type":"ContainerDied","Data":"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d"} Jul 20 13:33:48.148426 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.148357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-77ntv" event={"ID":"7366c820-9c47-4d7d-ba14-900dd5cc3c35","Type":"ContainerDied","Data":"cb5f02b89f1ae388109fc9b6e26b547b6459c3ada1095ace8ed7db1eeb25d0cb"} Jul 20 13:33:48.148426 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.148383 2577 scope.go:117] "RemoveContainer" containerID="49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d" Jul 20 13:33:48.159313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.159296 2577 scope.go:117] "RemoveContainer" containerID="49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d" Jul 20 13:33:48.159532 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:33:48.159516 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d\": container with ID starting with 49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d not found: ID does not exist" containerID="49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d" Jul 20 13:33:48.159591 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.159545 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d"} err="failed to get container status \"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d\": rpc error: code = NotFound desc = could not find container \"49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d\": container with ID starting with 49c0b5a753f00c609a90e4694d1909fbbf902403fb6c57e378afb37d2b82e55d not found: ID does not exist" Jul 20 13:33:48.171841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.171816 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:33:48.175906 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:48.175883 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-77ntv"] Jul 20 13:33:49.394321 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:33:49.394288 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" path="/var/lib/kubelet/pods/7366c820-9c47-4d7d-ba14-900dd5cc3c35/volumes" Jul 20 13:35:06.843197 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:06.843124 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:08.059147 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:08.059072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:48.891505 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:48.891473 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:48.993390 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:48.993359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:49.108355 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:49.108325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:55.205150 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:55.205114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:55.314733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:55.314704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:55.408014 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:55.407978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:35:55.563775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:35:55.563707 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:36:01.629573 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:01.629534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:36:01.701427 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:01.701390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:36:47.225820 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:47.225793 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:36:47.228883 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:47.228851 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:36:47.232456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:47.232437 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:36:47.235465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:36:47.235443 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:37:56.296231 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:37:56.296192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:39:00.080030 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:00.079992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:39:56.523728 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:56.523692 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:39:56.524276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:56.523931 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" podUID="0f822d67-92f4-4392-8fb4-b16b319902c6" containerName="manager" containerID="cri-o://28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542" gracePeriod=10 Jul 20 13:39:59.488276 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.488250 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:39:59.571096 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.571016 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume\") pod \"0f822d67-92f4-4392-8fb4-b16b319902c6\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " Jul 20 13:39:59.571265 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.571152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbt8c\" (UniqueName: \"kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c\") pod \"0f822d67-92f4-4392-8fb4-b16b319902c6\" (UID: \"0f822d67-92f4-4392-8fb4-b16b319902c6\") " Jul 20 13:39:59.571463 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.571436 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f822d67-92f4-4392-8fb4-b16b319902c6" (UID: "0f822d67-92f4-4392-8fb4-b16b319902c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:39:59.573052 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.573032 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c" (OuterVolumeSpecName: "kube-api-access-lbt8c") pod "0f822d67-92f4-4392-8fb4-b16b319902c6" (UID: "0f822d67-92f4-4392-8fb4-b16b319902c6"). InnerVolumeSpecName "kube-api-access-lbt8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:39:59.672421 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.672388 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbt8c\" (UniqueName: \"kubernetes.io/projected/0f822d67-92f4-4392-8fb4-b16b319902c6-kube-api-access-lbt8c\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:39:59.672421 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:39:59.672417 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f822d67-92f4-4392-8fb4-b16b319902c6-extensions-socket-volume\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:40:00.313799 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.313764 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f822d67-92f4-4392-8fb4-b16b319902c6" containerID="28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542" exitCode=0 Jul 20 13:40:00.313971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.313822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" event={"ID":"0f822d67-92f4-4392-8fb4-b16b319902c6","Type":"ContainerDied","Data":"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542"} Jul 20 13:40:00.317207 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.314201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" Jul 20 13:40:00.317207 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.314291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq" event={"ID":"0f822d67-92f4-4392-8fb4-b16b319902c6","Type":"ContainerDied","Data":"48e87a27bc193fca644f4ce1f63a5e013f7275778a4bcb017f24d86a008a9be3"} Jul 20 13:40:00.317207 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.314359 2577 scope.go:117] "RemoveContainer" containerID="28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542" Jul 20 13:40:00.331801 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.331784 2577 scope.go:117] "RemoveContainer" containerID="28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542" Jul 20 13:40:00.332055 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:40:00.332036 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542\": container with ID starting with 28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542 not found: ID does not exist" containerID="28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542" Jul 20 13:40:00.332136 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.332068 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542"} err="failed to get container status \"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542\": rpc error: code = NotFound desc = could not find container \"28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542\": container with ID starting with 28bc77ff1c1199c283d09ca99bb12165681fb5280435a1cfba0213d18f017542 not found: ID does not exist" Jul 20 13:40:00.340589 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.340566 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:40:00.344978 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:00.344958 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lx6nq"] Jul 20 13:40:01.393594 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:01.393558 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f822d67-92f4-4392-8fb4-b16b319902c6" path="/var/lib/kubelet/pods/0f822d67-92f4-4392-8fb4-b16b319902c6/volumes" Jul 20 13:40:57.734654 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.734620 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb"] Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735056 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f822d67-92f4-4392-8fb4-b16b319902c6" containerName="manager" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735068 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f822d67-92f4-4392-8fb4-b16b319902c6" containerName="manager" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735080 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" containerName="authorino" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735085 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" containerName="authorino" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735095 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" containerName="manager" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735102 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" containerName="manager" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d35ce2-ada9-4ac7-b066-aea33bb25d1c" containerName="authorino" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735212 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f822d67-92f4-4392-8fb4-b16b319902c6" containerName="manager" Jul 20 13:40:57.736971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.735221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7366c820-9c47-4d7d-ba14-900dd5cc3c35" containerName="manager" Jul 20 13:40:57.738225 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.738210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.740676 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.740654 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vdzsh\"" Jul 20 13:40:57.750995 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.750972 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb"] Jul 20 13:40:57.887456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.887421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de09fb1-8af3-45c7-96ab-626d6f6f485e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.887624 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.887521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kcd9\" (UniqueName: \"kubernetes.io/projected/1de09fb1-8af3-45c7-96ab-626d6f6f485e-kube-api-access-2kcd9\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.988331 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.988245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kcd9\" (UniqueName: \"kubernetes.io/projected/1de09fb1-8af3-45c7-96ab-626d6f6f485e-kube-api-access-2kcd9\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.988331 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.988325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de09fb1-8af3-45c7-96ab-626d6f6f485e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.988744 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.988719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de09fb1-8af3-45c7-96ab-626d6f6f485e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:57.999646 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:57.999618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kcd9\" (UniqueName: \"kubernetes.io/projected/1de09fb1-8af3-45c7-96ab-626d6f6f485e-kube-api-access-2kcd9\") pod \"kuadrant-operator-controller-manager-86878c769f-8qpqb\" (UID: \"1de09fb1-8af3-45c7-96ab-626d6f6f485e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:58.048753 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.048726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:58.397325 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.397298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb"] Jul 20 13:40:58.397784 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:40:58.397762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de09fb1_8af3_45c7_96ab_626d6f6f485e.slice/crio-64401c8b6feb6069cfab7c389a5a3b3abcd33a5ffbefb1042a8366e914a3dd1f WatchSource:0}: Error finding container 64401c8b6feb6069cfab7c389a5a3b3abcd33a5ffbefb1042a8366e914a3dd1f: Status 404 returned error can't find the container with id 64401c8b6feb6069cfab7c389a5a3b3abcd33a5ffbefb1042a8366e914a3dd1f Jul 20 13:40:58.647926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.647834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" event={"ID":"1de09fb1-8af3-45c7-96ab-626d6f6f485e","Type":"ContainerStarted","Data":"40e38b7bb40bfa5f7598f4be3a8e910e71a5b7ee9eb085ed3728292db8f8b0a4"} Jul 20 13:40:58.648077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.647994 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:40:58.648077 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.648038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" event={"ID":"1de09fb1-8af3-45c7-96ab-626d6f6f485e","Type":"ContainerStarted","Data":"64401c8b6feb6069cfab7c389a5a3b3abcd33a5ffbefb1042a8366e914a3dd1f"} Jul 20 13:40:58.668306 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:40:58.668259 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" podStartSLOduration=1.668245306 podStartE2EDuration="1.668245306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:40:58.667531151 +0000 UTC m=+1457.731880059" watchObservedRunningTime="2026-07-20 13:40:58.668245306 +0000 UTC m=+1457.732594221" Jul 20 13:41:09.656154 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:41:09.656126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8qpqb" Jul 20 13:41:47.263561 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:41:47.263527 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:41:47.269845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:41:47.269821 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:41:47.269985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:41:47.269824 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:41:47.276470 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:41:47.276453 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:42:46.012113 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:42:46.012062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:42:51.872071 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:42:51.872037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:43:42.509892 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:43:42.509808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:43:49.659791 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:43:49.659735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:44:24.967339 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:44:24.967279 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:44:25.807841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:44:25.807779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:46:47.302558 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:47.302529 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:46:47.309041 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:47.309014 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:46:47.311759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:47.311731 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:46:47.317641 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:47.317624 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:46:56.156306 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:56.156265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:46:56.245902 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:46:56.245857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:47:33.604373 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:47:33.604339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:47:52.393439 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:47:52.393401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:47:59.446072 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:47:59.446034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:11.336043 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:11.335960 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:19.616250 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:19.616207 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:54.871857 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:54.871805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:57.924056 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:57.924015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:58.016895 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:58.016855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:48:58.374735 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:48:58.374704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:04.406045 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:04.406009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:04.514664 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:04.514621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:04.722165 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:04.722078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:04.948016 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:04.947981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:05.544259 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:05.544226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:06.981278 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:06.981242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:07.077140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:07.077106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:07.148674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:07.148644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:07.249920 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:07.249826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:07.649137 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:07.649092 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:07.749798 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:07.749766 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:43.175411 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:43.175309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:47.650204 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:47.650135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:47.778593 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:47.778545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.017264 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.017179 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.293057 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.292967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.391304 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.391257 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.492681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.492647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.786748 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.786717 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:49:48.790457 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.790428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.793319 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.793297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5dcc91bd-gw-openshift-default-dockercfg-4nzm9\"" Jul 20 13:49:48.804403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.804383 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:49:48.814661 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.814634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:48.863482 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863589 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863589 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863667 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863706 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863796 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.863832 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.863790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hlx\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965411 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965411 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965411 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hlx\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965595 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.965845 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.965821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.966025 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.966003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.966065 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.966010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.966101 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.966076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.967785 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.967757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.967891 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.967795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.973666 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.973642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:48.974143 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:48.974124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hlx\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx\") pod \"e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:49.080296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.080210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:49.103901 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.103869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:49.246270 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.246242 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:49:49.247556 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:49:49.247529 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a5668b_506f_49e2_a5f7_7577311e1ba6.slice/crio-900ae2d10af4c03e7a2a1b5d7a47789d770f9da2f8b6a3147e336853eb245903 WatchSource:0}: Error finding container 900ae2d10af4c03e7a2a1b5d7a47789d770f9da2f8b6a3147e336853eb245903: Status 404 returned error can't find the container with id 900ae2d10af4c03e7a2a1b5d7a47789d770f9da2f8b6a3147e336853eb245903 Jul 20 13:49:49.249678 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.249645 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:49:49.249788 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.249728 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:49:49.249788 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.249777 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:49:49.703560 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.703525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerStarted","Data":"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd"} Jul 20 13:49:49.703710 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.703579 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerStarted","Data":"900ae2d10af4c03e7a2a1b5d7a47789d770f9da2f8b6a3147e336853eb245903"} Jul 20 13:49:49.724839 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:49.724796 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podStartSLOduration=1.724781956 podStartE2EDuration="1.724781956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:49:49.72296535 +0000 UTC m=+1988.787314268" watchObservedRunningTime="2026-07-20 13:49:49.724781956 +0000 UTC m=+1988.789130871" Jul 20 13:49:50.105003 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:50.104927 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:50.109844 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:50.109820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:50.710902 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:50.710875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:50.711828 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:50.711809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:49:51.189114 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:51.189080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:51.497708 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:51.497615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:52.077397 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:52.077346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:49:52.177365 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:49:52.177332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:02.781540 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.781506 2577 generic.go:356] "Generic (PLEG): container finished" podID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerID="babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd" exitCode=0 Jul 20 13:50:02.781971 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.781582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerDied","Data":"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd"} Jul 20 13:50:02.782092 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.782075 2577 scope.go:117] "RemoveContainer" containerID="babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd" Jul 20 13:50:02.783107 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.783081 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:50:02.783159 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.783152 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:50:02.783219 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:02.783199 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:50:03.790567 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:03.790525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerStarted","Data":"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df"} Jul 20 13:50:04.104536 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:04.104422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:04.106015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:04.105984 2577 patch_prober.go:28] interesting pod/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 20 13:50:04.106247 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:04.106222 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 20 13:50:05.105022 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:05.104975 2577 patch_prober.go:28] interesting pod/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 20 13:50:05.105557 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:05.105064 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 20 13:50:05.711299 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:05.711252 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:06.104734 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:06.104637 2577 patch_prober.go:28] interesting pod/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 20 13:50:06.104988 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:06.104960 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 20 13:50:07.104987 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:07.104939 2577 patch_prober.go:28] interesting pod/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 20 13:50:07.105526 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:07.105016 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 20 13:50:08.108015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:08.107981 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:08.108922 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:08.108903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:26.868713 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:26.868671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:30.405107 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:30.405068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:30.504585 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:30.504520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:31.325133 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.325092 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:31.474483 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.474442 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:50:31.474859 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.474709 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" containerID="cri-o://3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df" gracePeriod=30 Jul 20 13:50:31.489559 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.489531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:31.725750 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.725708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:31.948497 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:31.948463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:32.298657 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:32.298623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:32.446320 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:32.446289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:34.393775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:34.393738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:34.480770 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:34.480734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:34.547214 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:34.547165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:34.892336 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:34.892301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:35.232961 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:35.232885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:35.339100 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:35.339062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:50:36.739212 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.739142 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:36.797273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797273 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797267 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797288 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797355 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797381 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88hlx\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797520 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797444 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.797779 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797567 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:50:36.797976 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797944 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:50:36.797976 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.797968 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.798106 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798038 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket\") pod \"99a5668b-506f-49e2-a5f7-7577311e1ba6\" (UID: \"99a5668b-506f-49e2-a5f7-7577311e1ba6\") " Jul 20 13:50:36.798235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798197 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:50:36.798356 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798336 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:50:36.798467 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798449 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.798527 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798475 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a5668b-506f-49e2-a5f7-7577311e1ba6-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.798527 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798490 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.798527 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798503 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.798697 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.798536 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data" (OuterVolumeSpecName: "istio-data") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:50:36.800383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.800356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx" (OuterVolumeSpecName: "kube-api-access-88hlx") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "kube-api-access-88hlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:50:36.800383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.800365 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:50:36.800383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.800356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token" (OuterVolumeSpecName: "istio-token") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:50:36.800630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.800512 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99a5668b-506f-49e2-a5f7-7577311e1ba6" (UID: "99a5668b-506f-49e2-a5f7-7577311e1ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:50:36.899073 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.899017 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.899073 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.899064 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hlx\" (UniqueName: \"kubernetes.io/projected/99a5668b-506f-49e2-a5f7-7577311e1ba6-kube-api-access-88hlx\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.899073 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.899079 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.899373 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.899092 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.899373 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.899104 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a5668b-506f-49e2-a5f7-7577311e1ba6-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:50:36.999601 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.999511 2577 generic.go:356] "Generic (PLEG): container finished" podID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerID="3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df" exitCode=0 Jul 20 13:50:36.999601 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.999575 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" Jul 20 13:50:36.999778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.999609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerDied","Data":"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df"} Jul 20 13:50:36.999778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.999661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j" event={"ID":"99a5668b-506f-49e2-a5f7-7577311e1ba6","Type":"ContainerDied","Data":"900ae2d10af4c03e7a2a1b5d7a47789d770f9da2f8b6a3147e336853eb245903"} Jul 20 13:50:36.999778 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:36.999685 2577 scope.go:117] "RemoveContainer" containerID="3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df" Jul 20 13:50:37.010378 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.010364 2577 scope.go:117] "RemoveContainer" containerID="babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd" Jul 20 13:50:37.018682 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.018663 2577 scope.go:117] "RemoveContainer" containerID="3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df" Jul 20 13:50:37.018953 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:50:37.018932 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df\": container with ID starting with 3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df not found: ID does not exist" containerID="3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df" Jul 20 13:50:37.019024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.018961 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df"} err="failed to get container status \"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df\": rpc error: code = NotFound desc = could not find container \"3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df\": container with ID starting with 3b5d051994f02ffd9016614856bf58dc8cfc3adb50e914fcee901927ab0044df not found: ID does not exist" Jul 20 13:50:37.019024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.018979 2577 scope.go:117] "RemoveContainer" containerID="babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd" Jul 20 13:50:37.019225 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:50:37.019207 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd\": container with ID starting with babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd not found: ID does not exist" containerID="babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd" Jul 20 13:50:37.019289 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.019236 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd"} err="failed to get container status \"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd\": rpc error: code = NotFound desc = could not find container \"babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd\": container with ID starting with babf4d2107153edc96deb5022e9eb3d870e18d8f793f3f6274558b76722838dd not found: ID does not exist" Jul 20 13:50:37.024250 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.024159 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:50:37.027799 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.027780 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5dcc91bd-gw-openshift-default-6f867dd98c-ddh2j"] Jul 20 13:50:37.394084 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:50:37.394050 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" path="/var/lib/kubelet/pods/99a5668b-506f-49e2-a5f7-7577311e1ba6/volumes" Jul 20 13:51:10.793376 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:10.793271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:15.359038 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:15.358953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:16.741545 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:16.741512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:16.963967 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:16.963924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:17.229210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.229146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:17.614570 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.614541 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:51:17.614970 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.614958 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.615015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.614972 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.615015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.614985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.615015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.614991 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.615110 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.615055 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.615110 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.615064 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a5668b-506f-49e2-a5f7-7577311e1ba6" containerName="istio-proxy" Jul 20 13:51:17.618129 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.618113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.621005 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.620989 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b79b63cc-openshift-default-dockercfg-d8rfm\"" Jul 20 13:51:17.642246 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.642222 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:51:17.658807 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.658763 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:17.664493 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664624 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664624 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664624 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664786 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664786 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrqb\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664786 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664928 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.664928 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.664849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765368 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765368 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrqb\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.765865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.765784 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.766327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.766128 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.766327 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.766233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.766435 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.766360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.766720 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.766696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.767863 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.767837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.768252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.768234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.774914 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.774891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.775417 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.775397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrqb\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb\") pod \"e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:17.879494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.879410 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:17.928283 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:17.928251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:18.077878 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.077848 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:51:18.079553 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:51:18.079529 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4395972_3493_42ba_929d_b97823f62cc6.slice/crio-c4bbb9748bbb0f2fa9a7d77285a53a607690342c667f5b6fab92fc09022fdc4f WatchSource:0}: Error finding container c4bbb9748bbb0f2fa9a7d77285a53a607690342c667f5b6fab92fc09022fdc4f: Status 404 returned error can't find the container with id c4bbb9748bbb0f2fa9a7d77285a53a607690342c667f5b6fab92fc09022fdc4f Jul 20 13:51:18.082403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.082369 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:18.082502 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.082435 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:18.082502 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.082465 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:18.245268 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.245222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerStarted","Data":"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e"} Jul 20 13:51:18.245426 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.245288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerStarted","Data":"c4bbb9748bbb0f2fa9a7d77285a53a607690342c667f5b6fab92fc09022fdc4f"} Jul 20 13:51:18.272557 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.272488 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podStartSLOduration=1.272468095 podStartE2EDuration="1.272468095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:51:18.270054728 +0000 UTC m=+2077.334403644" watchObservedRunningTime="2026-07-20 13:51:18.272468095 +0000 UTC m=+2077.336817011" Jul 20 13:51:18.928418 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:18.928385 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:19.797901 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:19.797862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:19.920118 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:19.920088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:19.933473 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:19.933449 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:20.027014 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:20.026978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:20.259143 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:20.259112 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:20.260277 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:20.260252 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:20.415834 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:20.415800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:21.006941 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:21.006902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:21.123868 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:21.123830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:31.327988 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.327954 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4395972-3493-42ba-929d-b97823f62cc6" containerID="832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e" exitCode=0 Jul 20 13:51:31.334663 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.328203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerDied","Data":"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e"} Jul 20 13:51:31.335441 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.335420 2577 scope.go:117] "RemoveContainer" containerID="832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e" Jul 20 13:51:31.336674 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.336642 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:31.336734 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.336714 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:31.336775 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:31.336745 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:51:32.335954 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:32.335917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerStarted","Data":"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa"} Jul 20 13:51:32.929049 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:32.929017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:32.930264 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:32.930237 2577 patch_prober.go:28] interesting pod/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 20 13:51:32.930383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:32.930288 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 20 13:51:33.929154 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:33.929109 2577 patch_prober.go:28] interesting pod/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 20 13:51:33.929603 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:33.929184 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 20 13:51:34.929519 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:34.929479 2577 patch_prober.go:28] interesting pod/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 20 13:51:34.929937 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:34.929552 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 20 13:51:35.259745 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:35.259658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:35.928641 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:35.928603 2577 patch_prober.go:28] interesting pod/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 20 13:51:35.928830 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:35.928665 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 20 13:51:36.933583 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:36.933553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:36.934597 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:36.934566 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:51:47.349581 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:47.349554 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:51:47.355806 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:47.355782 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:51:47.358774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:47.358754 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:51:47.364294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:47.364275 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:51:55.145305 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:55.145269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:59.735841 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:59.735801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:51:59.871934 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:59.871896 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:51:59.872139 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:59.872115 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" containerID="cri-o://d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa" gracePeriod=30 Jul 20 13:51:59.896944 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:51:59.896881 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:00.195292 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:00.195255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:00.742293 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:00.742256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:00.841356 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:00.841303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:00.941859 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:00.941827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:02.651011 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:02.650975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:05.136653 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.136598 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:52:05.211031 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211000 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211045 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211086 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211252 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211208 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211282 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211322 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhrqb\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211318 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:52:05.211548 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211383 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket\") pod \"a4395972-3493-42ba-929d-b97823f62cc6\" (UID: \"a4395972-3493-42ba-929d-b97823f62cc6\") " Jul 20 13:52:05.211759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:52:05.211759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211620 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:52:05.211866 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:52:05.211926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211873 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.211926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211898 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.211926 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.211912 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4395972-3493-42ba-929d-b97823f62cc6-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.212620 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.212588 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:52:05.213492 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.213461 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:52:05.213589 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.213491 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:52:05.213678 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.213651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb" (OuterVolumeSpecName: "kube-api-access-hhrqb") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "kube-api-access-hhrqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:52:05.213826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.213806 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4395972-3493-42ba-929d-b97823f62cc6" (UID: "a4395972-3493-42ba-929d-b97823f62cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:52:05.312911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312885 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.312911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312907 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.312911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312916 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4395972-3493-42ba-929d-b97823f62cc6-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.313097 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312925 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.313097 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312933 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrqb\" (UniqueName: \"kubernetes.io/projected/a4395972-3493-42ba-929d-b97823f62cc6-kube-api-access-hhrqb\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.313097 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.312941 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4395972-3493-42ba-929d-b97823f62cc6-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:52:05.541569 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.541538 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4395972-3493-42ba-929d-b97823f62cc6" containerID="d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa" exitCode=0 Jul 20 13:52:05.541733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.541591 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" Jul 20 13:52:05.541733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.541617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerDied","Data":"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa"} Jul 20 13:52:05.541733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.541667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds" event={"ID":"a4395972-3493-42ba-929d-b97823f62cc6","Type":"ContainerDied","Data":"c4bbb9748bbb0f2fa9a7d77285a53a607690342c667f5b6fab92fc09022fdc4f"} Jul 20 13:52:05.541733 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.541689 2577 scope.go:117] "RemoveContainer" containerID="d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa" Jul 20 13:52:05.551138 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.551122 2577 scope.go:117] "RemoveContainer" containerID="832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e" Jul 20 13:52:05.558611 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.558584 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:52:05.559759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.559727 2577 scope.go:117] "RemoveContainer" containerID="d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa" Jul 20 13:52:05.560088 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:52:05.560069 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa\": container with ID starting with d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa not found: ID does not exist" containerID="d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa" Jul 20 13:52:05.560153 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.560095 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa"} err="failed to get container status \"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa\": rpc error: code = NotFound desc = could not find container \"d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa\": container with ID starting with d2d6556bf2d6e9edc4a8ef3d2b3e6fa97aa4fdc6d1a37131c7e90e9e35b4f2fa not found: ID does not exist" Jul 20 13:52:05.560153 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.560115 2577 scope.go:117] "RemoveContainer" containerID="832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e" Jul 20 13:52:05.560385 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:52:05.560366 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e\": container with ID starting with 832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e not found: ID does not exist" containerID="832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e" Jul 20 13:52:05.560454 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.560391 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e"} err="failed to get container status \"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e\": rpc error: code = NotFound desc = could not find container \"832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e\": container with ID starting with 832d593e423af68a991d0b8bb5725263dfdc658a2567c2ebd1df02332175e70e not found: ID does not exist" Jul 20 13:52:05.562825 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:05.562804 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b79b63cc-openshift-default-769f4b967f-2lcds"] Jul 20 13:52:07.394556 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:07.394516 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4395972-3493-42ba-929d-b97823f62cc6" path="/var/lib/kubelet/pods/a4395972-3493-42ba-929d-b97823f62cc6/volumes" Jul 20 13:52:09.053987 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:09.053951 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:09.161932 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:09.161894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:09.257429 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:09.257380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:09.355304 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:09.355209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:09.853478 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:09.853446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:30.603089 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:30.603048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:30.660865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:30.660831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:35.845649 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:35.845573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:42.049049 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:42.049005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:42.150586 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:42.150535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:42.251469 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:42.251435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:52:58.161999 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:52:58.161962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:04.099665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:04.099632 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:04.207342 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:04.207301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:04.301258 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:04.301220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:05.264332 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:05.264293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:14.845695 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:14.845654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:15.668101 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:15.668063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:21.798084 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:21.798047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:21.909770 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:21.909737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:22.001836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:22.001799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:22.102009 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:22.101937 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:22.903318 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:22.903275 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:22.999682 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:22.999652 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:23.097846 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:23.097800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:24.712200 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:24.712149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:30.930072 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:30.930033 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:31.030857 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:31.030810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:31.621325 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:31.621288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:37.452028 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:37.451989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:37.554961 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:37.554927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:37.701399 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:37.701349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:38.029715 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.029680 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:53:38.030248 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.030228 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:53:38.030248 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.030249 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:53:38.030371 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.030262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:53:38.030371 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.030267 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:53:38.030371 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.030334 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:53:38.033464 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.033447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.035865 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.035840 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ffcdeab1-openshift-default-dockercfg-frxdj\"" Jul 20 13:53:38.041788 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.041762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:38.055406 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.055385 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:53:38.149018 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.148984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149018 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149253 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149253 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149253 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149253 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5db\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.149383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.149316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250210 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5db\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250456 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250826 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.250994 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.250971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.251161 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.251140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.252754 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.252737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.252985 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.252962 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.258790 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.258765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5db\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.258885 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.258800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token\") pod \"e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.344702 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.344629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:38.395736 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.395697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:38.494294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.494265 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:53:38.495220 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:53:38.495189 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d97e08_084c_422e_ba76_e247d2c11785.slice/crio-bdc0da5983db1adacaced8c7dd1044d4241db039ebf1020a36b922b89be8f8dd WatchSource:0}: Error finding container bdc0da5983db1adacaced8c7dd1044d4241db039ebf1020a36b922b89be8f8dd: Status 404 returned error can't find the container with id bdc0da5983db1adacaced8c7dd1044d4241db039ebf1020a36b922b89be8f8dd Jul 20 13:53:38.498402 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.498360 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:38.498541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.498464 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:38.498541 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:38.498506 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:39.080213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:39.080150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerStarted","Data":"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c"} Jul 20 13:53:39.080422 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:39.080271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerStarted","Data":"bdc0da5983db1adacaced8c7dd1044d4241db039ebf1020a36b922b89be8f8dd"} Jul 20 13:53:39.102714 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:39.102651 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podStartSLOduration=1.102632749 podStartE2EDuration="1.102632749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:53:39.099151798 +0000 UTC m=+2218.163500713" watchObservedRunningTime="2026-07-20 13:53:39.102632749 +0000 UTC m=+2218.166981668" Jul 20 13:53:39.345891 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:39.345809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:39.351103 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:39.351079 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:40.088448 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:40.088422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:40.089373 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:40.089343 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:41.111719 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:41.111685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:41.210662 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:41.210629 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:41.348101 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:41.348050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:41.458715 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:41.458634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:41.908629 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:41.908599 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:42.008853 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:42.008814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:45.512079 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:45.512049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:46.187717 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:46.187679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:46.275008 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:46.274974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:46.611294 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:46.611256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:52.870328 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:52.870289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:52.970230 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:52.970154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:53.512864 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:53.512829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:53:54.175531 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.175496 2577 generic.go:356] "Generic (PLEG): container finished" podID="d9d97e08-084c-422e-ba76-e247d2c11785" containerID="e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c" exitCode=0 Jul 20 13:53:54.175947 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.175565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerDied","Data":"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c"} Jul 20 13:53:54.175999 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.175984 2577 scope.go:117] "RemoveContainer" containerID="e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c" Jul 20 13:53:54.177087 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.177060 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:54.177164 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.177129 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:54.177232 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:54.177165 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:53:55.088793 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:55.088748 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:55.194323 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:55.194285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerStarted","Data":"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55"} Jul 20 13:53:55.345848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:55.345759 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:55.346852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:55.346819 2577 patch_prober.go:28] interesting pod/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 20 13:53:55.346996 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:55.346884 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 20 13:53:56.345484 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:56.345447 2577 patch_prober.go:28] interesting pod/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 20 13:53:56.345849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:56.345501 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 20 13:53:57.345669 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:57.345634 2577 patch_prober.go:28] interesting pod/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 20 13:53:57.346081 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:57.345691 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 20 13:53:58.345145 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:58.345100 2577 patch_prober.go:28] interesting pod/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 20 13:53:58.345366 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:58.345165 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 20 13:53:59.349819 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:59.349790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:59.350308 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:59.350216 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:53:59.350998 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:53:59.350979 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:54:03.703461 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:03.703369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:08.303036 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:08.302971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:08.690481 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:08.690452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:14.460514 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:14.460477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:14.559896 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:14.559858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:14.717535 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:14.717445 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:54:14.717758 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:14.717729 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" containerID="cri-o://c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55" gracePeriod=30 Jul 20 13:54:14.742050 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:14.742019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:15.013600 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:15.013509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:15.148024 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:15.147972 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:15.243332 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:15.243304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:15.360270 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:15.360231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:19.965848 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:19.965827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:54:20.047739 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047654 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.047739 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn5db\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.047739 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047727 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047747 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047803 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047848 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047873 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047898 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.047920 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data\") pod \"d9d97e08-084c-422e-ba76-e247d2c11785\" (UID: \"d9d97e08-084c-422e-ba76-e247d2c11785\") " Jul 20 13:54:20.048360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:54:20.048360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048289 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:54:20.048360 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:54:20.048521 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048349 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.048651 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048621 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:54:20.048919 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.048898 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:54:20.049916 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.049890 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db" (OuterVolumeSpecName: "kube-api-access-wn5db") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "kube-api-access-wn5db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:54:20.050010 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.049957 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:54:20.050376 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.050350 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:54:20.050532 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.050514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9d97e08-084c-422e-ba76-e247d2c11785" (UID: "d9d97e08-084c-422e-ba76-e247d2c11785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:54:20.149574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149538 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d97e08-084c-422e-ba76-e247d2c11785-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149565 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5db\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-kube-api-access-wn5db\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149575 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d97e08-084c-422e-ba76-e247d2c11785-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149584 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149592 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149599 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d97e08-084c-422e-ba76-e247d2c11785-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149608 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.149836 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.149615 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d97e08-084c-422e-ba76-e247d2c11785-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 13:54:20.358227 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.358129 2577 generic.go:356] "Generic (PLEG): container finished" podID="d9d97e08-084c-422e-ba76-e247d2c11785" containerID="c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55" exitCode=0 Jul 20 13:54:20.358227 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.358209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerDied","Data":"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55"} Jul 20 13:54:20.358491 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.358291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" event={"ID":"d9d97e08-084c-422e-ba76-e247d2c11785","Type":"ContainerDied","Data":"bdc0da5983db1adacaced8c7dd1044d4241db039ebf1020a36b922b89be8f8dd"} Jul 20 13:54:20.358491 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.358218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244" Jul 20 13:54:20.358491 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.358313 2577 scope.go:117] "RemoveContainer" containerID="c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55" Jul 20 13:54:20.369374 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.369353 2577 scope.go:117] "RemoveContainer" containerID="e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c" Jul 20 13:54:20.377769 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.377745 2577 scope.go:117] "RemoveContainer" containerID="c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55" Jul 20 13:54:20.378063 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:54:20.378044 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55\": container with ID starting with c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55 not found: ID does not exist" containerID="c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55" Jul 20 13:54:20.378114 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.378072 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55"} err="failed to get container status \"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55\": rpc error: code = NotFound desc = could not find container \"c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55\": container with ID starting with c5da434fff17fcf8cbd5c1bd6ac554b18f7d36685a876e52f23a1a219d252f55 not found: ID does not exist" Jul 20 13:54:20.378114 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.378090 2577 scope.go:117] "RemoveContainer" containerID="e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c" Jul 20 13:54:20.378342 ip-10-0-141-54 kubenswrapper[2577]: E0720 13:54:20.378326 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c\": container with ID starting with e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c not found: ID does not exist" containerID="e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c" Jul 20 13:54:20.378389 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.378348 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c"} err="failed to get container status \"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c\": rpc error: code = NotFound desc = could not find container \"e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c\": container with ID starting with e3dfe8c8fe8488fdccda93f0e31eb3b4fcffc1a5c64f1aa3959e50019ae1d44c not found: ID does not exist" Jul 20 13:54:20.383920 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.383893 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:54:20.386653 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:20.386633 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ffcdeab1-openshift-default-6cc85c8bf5-p9244"] Jul 20 13:54:21.077208 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:21.077147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:21.180888 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:21.180859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:21.277578 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:21.277545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:21.393894 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:21.393866 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" path="/var/lib/kubelet/pods/d9d97e08-084c-422e-ba76-e247d2c11785/volumes" Jul 20 13:54:21.456140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:21.456097 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:27.343311 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:27.343264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:27.444101 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:27.444069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:34.383188 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:34.383145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:34.765006 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:34.764927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:34.965911 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:34.965879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:38.206313 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:38.206277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:38.304097 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:38.304060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:38.420441 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:38.420395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:38.518774 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:38.518692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:38.914385 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:38.914352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:39.018333 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:39.018290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:42.718396 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:42.718355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:54:48.671213 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:54:48.671148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:26.485346 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:26.485309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:31.032318 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:31.032277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:31.348534 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:31.348430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:31.449702 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:31.449667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:33.360220 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:33.360125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:55:38.391759 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:55:38.391719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:14.494235 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:14.494196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:15.932221 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:15.932165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:21.683632 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:21.683597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:34.638370 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:34.638333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:34.759685 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:34.759651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:34.931633 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:34.931552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:35.179395 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:35.179359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:36.377044 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:36.377010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:36.495942 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:36.495897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:37.323367 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:37.323331 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:37.421744 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:37.421706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:37.523875 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:37.523844 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:38.931821 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:38.931779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:45.300906 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:45.300868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:45.401877 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:45.401843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:46.002186 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:46.002143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:47.391590 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:47.391564 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:56:47.399382 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:47.399350 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:56:47.403107 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:47.403085 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 13:56:47.409554 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:47.409535 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 13:56:52.404356 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:52.404323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:52.513145 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:52.513109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:56:52.604562 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:56:52.604513 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:03.217479 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:03.217384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:03.591140 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:03.591046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:22.223266 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:22.223230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:22.324547 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:22.324506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:24.620075 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:24.620043 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:30.550211 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:30.550146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:30.657702 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:30.657658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:30.982265 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:30.982233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:31.268403 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:31.268312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:31.643630 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:31.643594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.118406 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.118371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.219272 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.219227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.312888 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.312854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.626890 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.626861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.786567 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.786532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:34.899833 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:34.899743 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:35.492060 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:35.492028 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:38.648530 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:38.648492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:39.019404 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.019303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:39.320324 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320245 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 13:57:39.320758 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320740 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.320854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320763 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.320854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320776 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.320854 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320783 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.321063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320900 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4395972-3493-42ba-929d-b97823f62cc6" containerName="istio-proxy" Jul 20 13:57:39.321063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320917 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.321063 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.320928 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d97e08-084c-422e-ba76-e247d2c11785" containerName="istio-proxy" Jul 20 13:57:39.324129 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.324109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.327142 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.327117 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1af9e1-openshift-default-dockercfg-6glqf\"" Jul 20 13:57:39.334952 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.334929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 13:57:39.335083 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335152 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335286 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8kk\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335338 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335425 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335493 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335623 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.335692 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.335677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.341802 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.341778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:39.436439 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436439 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436665 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8kk\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.436975 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.436852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.437244 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.437218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.437348 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.437286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.437414 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.437378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.437465 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.437410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.438769 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.438750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.439015 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.438997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.444232 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.444207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.444351 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.444335 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8kk\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk\") pod \"e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.636744 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.636705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:39.778223 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.778189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 13:57:39.778978 ip-10-0-141-54 kubenswrapper[2577]: W0720 13:57:39.778947 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b28edfe_975c_413d_a54b_319c5889dec8.slice/crio-5267b780973333b44fb3681fe68983c57b1f2118ad9c5947dc85920c04df85e6 WatchSource:0}: Error finding container 5267b780973333b44fb3681fe68983c57b1f2118ad9c5947dc85920c04df85e6: Status 404 returned error can't find the container with id 5267b780973333b44fb3681fe68983c57b1f2118ad9c5947dc85920c04df85e6 Jul 20 13:57:39.781216 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.781158 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:39.781296 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.781270 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:39.781334 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.781319 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:39.909790 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:39.909698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:40.495040 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:40.494943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerStarted","Data":"fe962a04998f058a96aeb8df70347bf6ea12bd8e0112bdd845a5a7a186330d9b"} Jul 20 13:57:40.495040 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:40.495013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerStarted","Data":"5267b780973333b44fb3681fe68983c57b1f2118ad9c5947dc85920c04df85e6"} Jul 20 13:57:40.515904 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:40.515851 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podStartSLOduration=1.515836532 podStartE2EDuration="1.515836532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:57:40.513818883 +0000 UTC m=+2459.578167798" watchObservedRunningTime="2026-07-20 13:57:40.515836532 +0000 UTC m=+2459.580185446" Jul 20 13:57:40.637871 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:40.637832 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:40.642529 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:40.642503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:41.501892 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:41.501860 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:41.502812 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:41.502787 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:42.482328 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:42.482285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:42.581805 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:42.581773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:42.677527 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:42.677491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:43.359217 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:43.359158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:43.460113 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:43.460075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:47.054394 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:47.054358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:54.577822 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.577788 2577 generic.go:356] "Generic (PLEG): container finished" podID="4b28edfe-975c-413d-a54b-319c5889dec8" containerID="fe962a04998f058a96aeb8df70347bf6ea12bd8e0112bdd845a5a7a186330d9b" exitCode=0 Jul 20 13:57:54.578310 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.577836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerDied","Data":"fe962a04998f058a96aeb8df70347bf6ea12bd8e0112bdd845a5a7a186330d9b"} Jul 20 13:57:54.578310 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.578224 2577 scope.go:117] "RemoveContainer" containerID="fe962a04998f058a96aeb8df70347bf6ea12bd8e0112bdd845a5a7a186330d9b" Jul 20 13:57:54.579383 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.579343 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:54.579460 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.579415 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:54.579508 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.579476 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 13:57:54.742999 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:54.742961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:55.585574 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.585534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerStarted","Data":"dbe7285ea0930cdef4c44799240e3b97f3b634ce607c11cbd2242ee3616f144f"} Jul 20 13:57:55.637190 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.637134 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:55.638532 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.638505 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 20 13:57:55.638656 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.638569 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 20 13:57:55.736997 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.736960 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:55.835494 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:55.835454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:57:56.501771 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:56.501730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:56.637148 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:56.637107 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 20 13:57:56.637632 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:56.637162 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 20 13:57:57.638033 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:57.637999 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 20 13:57:57.638472 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:57.638061 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 20 13:57:58.637503 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:58.637468 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 20 13:57:58.637681 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:58.637520 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 20 13:57:59.641849 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:59.641821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:57:59.642852 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:57:59.642831 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 13:58:04.257941 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:04.257909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:58:04.358104 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:04.358070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:58:36.592946 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:36.592850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:58:38.651128 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:38.651092 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:58:38.708372 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:38.708334 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:58:57.881625 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:58:57.881590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:06.498738 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:06.498701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:32.971947 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:32.971907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:33.886887 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:33.886851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:34.557234 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:34.557201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:34.657297 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:34.657264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:44.718427 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:44.718376 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:49.126614 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:49.126578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:49.532119 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:49.532026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:55.615893 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:55.615856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:55.700019 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:55.699981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:55.952544 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:55.952446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:56.227870 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:56.227786 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:57.058344 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:57.058306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 13:59:57.162428 ip-10-0-141-54 kubenswrapper[2577]: I0720 13:59:57.162397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:07.020811 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:07.020727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:11.457521 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:11.457490 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:11.878451 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:11.878405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:18.060052 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.060011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:18.142643 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.142588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:18.308670 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.308629 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 14:00:18.308964 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.308936 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" containerID="cri-o://dbe7285ea0930cdef4c44799240e3b97f3b634ce607c11cbd2242ee3616f144f" gracePeriod=30 Jul 20 14:00:18.327541 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.327459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:18.555208 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.555156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:18.910952 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:18.910919 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:19.235394 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:19.235288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:19.809242 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:19.809209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:22.053971 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:22.053939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:22.339033 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:22.338934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:22.471842 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:22.471804 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:22.700921 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:22.700878 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:22.983677 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:22.983591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:23.085556 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.085523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:23.467289 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.467257 2577 generic.go:356] "Generic (PLEG): container finished" podID="4b28edfe-975c-413d-a54b-319c5889dec8" containerID="dbe7285ea0930cdef4c44799240e3b97f3b634ce607c11cbd2242ee3616f144f" exitCode=0 Jul 20 14:00:23.467443 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.467308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerDied","Data":"dbe7285ea0930cdef4c44799240e3b97f3b634ce607c11cbd2242ee3616f144f"} Jul 20 14:00:23.467443 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.467369 2577 scope.go:117] "RemoveContainer" containerID="fe962a04998f058a96aeb8df70347bf6ea12bd8e0112bdd845a5a7a186330d9b" Jul 20 14:00:23.570794 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.570733 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 14:00:23.674900 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.674865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.674921 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.674949 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.674965 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.675086 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675081 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.676236 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675108 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv8kk\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.676236 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675232 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket\") pod \"4b28edfe-975c-413d-a54b-319c5889dec8\" (UID: \"4b28edfe-975c-413d-a54b-319c5889dec8\") " Jul 20 14:00:23.676236 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:23.676236 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.675845 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.676236 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.676107 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:23.676460 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.676269 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:23.676672 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.676649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:00:23.676911 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.676889 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:23.677833 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.677796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:23.677914 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.677892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk" (OuterVolumeSpecName: "kube-api-access-tv8kk") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "kube-api-access-tv8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:23.677961 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.677888 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:23.678048 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.678018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b28edfe-975c-413d-a54b-319c5889dec8" (UID: "4b28edfe-975c-413d-a54b-319c5889dec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:00:23.776943 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776904 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b28edfe-975c-413d-a54b-319c5889dec8-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.776943 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776936 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b28edfe-975c-413d-a54b-319c5889dec8-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.776943 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776945 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.777199 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776954 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8kk\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-kube-api-access-tv8kk\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.777199 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776966 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.777199 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776974 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.777199 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776982 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b28edfe-975c-413d-a54b-319c5889dec8-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:23.777199 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:23.776990 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b28edfe-975c-413d-a54b-319c5889dec8-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:24.475337 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:24.475300 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" Jul 20 14:00:24.475337 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:24.475319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw" event={"ID":"4b28edfe-975c-413d-a54b-319c5889dec8","Type":"ContainerDied","Data":"5267b780973333b44fb3681fe68983c57b1f2118ad9c5947dc85920c04df85e6"} Jul 20 14:00:24.475834 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:24.475396 2577 scope.go:117] "RemoveContainer" containerID="dbe7285ea0930cdef4c44799240e3b97f3b634ce607c11cbd2242ee3616f144f" Jul 20 14:00:24.498024 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:24.497997 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 14:00:24.501952 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:24.501929 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1af9e1-openshift-default-56c6c967b9-q9wkw"] Jul 20 14:00:25.395140 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:25.395103 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" path="/var/lib/kubelet/pods/4b28edfe-975c-413d-a54b-319c5889dec8/volumes" Jul 20 14:00:26.406942 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:26.406907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:26.867209 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:26.867150 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:27.151734 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.151649 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:00:27.152228 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.152205 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:27.152228 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.152226 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:27.152361 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.152243 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:27.152361 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.152249 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:27.152361 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.152323 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:27.156775 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.156755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.159700 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.159683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ad282e-openshift-default-dockercfg-g8l24\"" Jul 20 14:00:27.171626 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.171440 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:00:27.180384 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.180357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:27.309897 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.309858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310085 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.309906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310085 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.309986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310085 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310085 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310279 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310279 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcks\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310279 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.310279 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.310255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411007 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.410920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411007 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.410958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411007 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.410995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcks\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411990 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411990 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.411990 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.411804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.413506 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.413486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.413633 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.413615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.419443 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.419417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.419571 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.419555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcks\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks\") pod \"e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.467545 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.467513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:27.618310 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.618280 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:00:27.618896 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:00:27.618868 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc408c80_9210_425d_938b_07a54f49cb40.slice/crio-83281d50d952204e2ad59a2ee7dbb92104b5ef98ae73d88f8f0e1d056cf4e9ed WatchSource:0}: Error finding container 83281d50d952204e2ad59a2ee7dbb92104b5ef98ae73d88f8f0e1d056cf4e9ed: Status 404 returned error can't find the container with id 83281d50d952204e2ad59a2ee7dbb92104b5ef98ae73d88f8f0e1d056cf4e9ed Jul 20 14:00:27.621127 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.621090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:27.621331 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.621302 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:27.621385 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.621370 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:27.621420 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:27.621398 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:28.500553 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:28.500515 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerStarted","Data":"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed"} Jul 20 14:00:28.501035 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:28.500581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerStarted","Data":"83281d50d952204e2ad59a2ee7dbb92104b5ef98ae73d88f8f0e1d056cf4e9ed"} Jul 20 14:00:28.527091 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:28.527043 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podStartSLOduration=1.527026328 podStartE2EDuration="1.527026328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:28.526085461 +0000 UTC m=+2627.590434376" watchObservedRunningTime="2026-07-20 14:00:28.527026328 +0000 UTC m=+2627.591375243" Jul 20 14:00:29.468249 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:29.468212 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:29.473005 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:29.472981 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:29.506515 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:29.506482 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:29.507450 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:29.507431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:30.510939 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:30.510899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:30.609276 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:30.609145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:30.824558 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:30.824460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:31.132495 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:31.132461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:31.598606 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:31.598515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:31.689474 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:31.689421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:32.750362 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:32.750328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:32.982043 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:32.982000 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:34.518072 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:34.518025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:34.715598 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:34.715564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:35.077301 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:35.077271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:43.181334 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.181292 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq"] Jul 20 14:00:43.181838 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.181822 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b28edfe-975c-413d-a54b-319c5889dec8" containerName="istio-proxy" Jul 20 14:00:43.186428 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.186413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:43.189043 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.189017 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 14:00:43.193052 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.192729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq"] Jul 20 14:00:43.361700 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.361668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq\" (UID: \"f082b94f-6917-4b0f-b9af-8eb2a5d53426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:43.463140 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.463047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq\" (UID: \"f082b94f-6917-4b0f-b9af-8eb2a5d53426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:43.463772 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.463745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq\" (UID: \"f082b94f-6917-4b0f-b9af-8eb2a5d53426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:43.503995 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.503961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:43.652535 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.652503 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq"] Jul 20 14:00:43.654086 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:00:43.654054 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf082b94f_6917_4b0f_b9af_8eb2a5d53426.slice/crio-7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0 WatchSource:0}: Error finding container 7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0: Status 404 returned error can't find the container with id 7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0 Jul 20 14:00:43.656246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:43.656228 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 14:00:44.005614 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:44.005571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:44.103991 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:44.103959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:44.601756 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:44.601712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" event={"ID":"f082b94f-6917-4b0f-b9af-8eb2a5d53426","Type":"ContainerStarted","Data":"7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0"} Jul 20 14:00:45.549361 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:45.549315 2577 patch_prober.go:28] interesting pod/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 14:00:45.549546 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:45.549407 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 14:00:46.617276 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.617247 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc408c80-9210-425d-938b-07a54f49cb40" containerID="df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed" exitCode=0 Jul 20 14:00:46.617723 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.617324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerDied","Data":"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed"} Jul 20 14:00:46.617723 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.617718 2577 scope.go:117] "RemoveContainer" containerID="df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed" Jul 20 14:00:46.618836 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.618805 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:46.618951 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.618889 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:46.618951 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.618921 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:00:46.619122 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.619102 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" event={"ID":"f082b94f-6917-4b0f-b9af-8eb2a5d53426","Type":"ContainerStarted","Data":"ba3b68d2e7558640c480121a0ba0ee4aa7f34f75c3660d6f910f17c4ffac9b2b"} Jul 20 14:00:46.659912 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:46.659869 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" podStartSLOduration=1.360722073 podStartE2EDuration="3.659850219s" totalImagesPullingTime="2.299128146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:46.657792607 +0000 UTC m=+2645.722141526" watchObservedRunningTime="2026-07-20 14:00:46.659850219 +0000 UTC m=+2645.724199133" Jul 20 14:00:47.626125 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:47.626091 2577 generic.go:356] "Generic (PLEG): container finished" podID="f082b94f-6917-4b0f-b9af-8eb2a5d53426" containerID="ba3b68d2e7558640c480121a0ba0ee4aa7f34f75c3660d6f910f17c4ffac9b2b" exitCode=0 Jul 20 14:00:47.626567 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:47.626133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" event={"ID":"f082b94f-6917-4b0f-b9af-8eb2a5d53426","Type":"ContainerDied","Data":"ba3b68d2e7558640c480121a0ba0ee4aa7f34f75c3660d6f910f17c4ffac9b2b"} Jul 20 14:00:47.628062 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:47.628038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerStarted","Data":"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603"} Jul 20 14:00:48.468382 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.468349 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:48.468566 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.468542 2577 patch_prober.go:28] interesting pod/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 20 14:00:48.468616 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.468590 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 20 14:00:48.823406 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.823376 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:48.924572 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.924539 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca\") pod \"f082b94f-6917-4b0f-b9af-8eb2a5d53426\" (UID: \"f082b94f-6917-4b0f-b9af-8eb2a5d53426\") " Jul 20 14:00:48.924923 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:48.924899 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f082b94f-6917-4b0f-b9af-8eb2a5d53426" (UID: "f082b94f-6917-4b0f-b9af-8eb2a5d53426"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:00:49.026241 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.026126 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f082b94f-6917-4b0f-b9af-8eb2a5d53426-maas-api-service-ca\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:00:49.468842 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.468807 2577 patch_prober.go:28] interesting pod/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 20 14:00:49.469050 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.468861 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 20 14:00:49.641596 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.641568 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" Jul 20 14:00:49.641794 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.641605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq" event={"ID":"f082b94f-6917-4b0f-b9af-8eb2a5d53426","Type":"ContainerDied","Data":"7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0"} Jul 20 14:00:49.641794 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:49.641663 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a37b23188d6c1b8c6458b55d4efd20e4754462023cb0375db33efa4ed11e1b0" Jul 20 14:00:50.468918 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:50.468876 2577 patch_prober.go:28] interesting pod/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 20 14:00:50.469413 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:50.468936 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 20 14:00:51.471536 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:51.471505 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:51.471990 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:51.471778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:51.472431 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:51.472416 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:00:52.478095 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:52.478063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:52.579209 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:52.579159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:00:58.390407 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:00:58.390369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:03.301501 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:03.301463 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq"] Jul 20 14:01:03.306041 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:03.306015 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f1ced-f9bkq"] Jul 20 14:01:03.308394 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:03.308365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:03.394727 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:03.394698 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f082b94f-6917-4b0f-b9af-8eb2a5d53426" path="/var/lib/kubelet/pods/f082b94f-6917-4b0f-b9af-8eb2a5d53426/volumes" Jul 20 14:01:03.687700 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:03.687658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:09.632913 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:09.632875 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:09.732856 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:09.732815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:09.904737 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:09.904650 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:10.125093 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:10.125060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:25.777121 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:25.777086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:30.382146 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:30.382104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:30.857785 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:30.857705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:37.694803 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:37.694713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:37.850501 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:37.850462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:38.020593 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:38.020509 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:01:38.020737 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:38.020704 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" containerID="cri-o://fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603" gracePeriod=30 Jul 20 14:01:38.064160 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:38.064125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:38.296726 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:38.296639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:38.682932 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:38.682899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:39.079697 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:39.079605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:39.352730 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:39.352642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:42.384389 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:42.384353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:42.506549 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:42.506509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:42.741012 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:42.740924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:42.840166 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:42.840123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:43.182216 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.182162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:43.295025 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.294963 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:01:43.341213 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341381 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341381 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341286 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341381 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341302 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341381 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341339 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341381 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341366 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341658 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341409 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341658 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341435 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jcks\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341658 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341472 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo\") pod \"bc408c80-9210-425d-938b-07a54f49cb40\" (UID: \"bc408c80-9210-425d-938b-07a54f49cb40\") " Jul 20 14:01:43.341658 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341627 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:43.341854 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:43.341854 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341653 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:43.341854 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.341665 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:01:43.342062 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.342029 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.342127 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.342070 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408c80-9210-425d-938b-07a54f49cb40-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.342127 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.342085 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.342127 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.342097 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.342376 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.342317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:43.343655 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.343633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:43.343840 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.343823 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:01:43.344151 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.344125 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks" (OuterVolumeSpecName: "kube-api-access-5jcks") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "kube-api-access-5jcks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:01:43.344151 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.344128 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc408c80-9210-425d-938b-07a54f49cb40" (UID: "bc408c80-9210-425d-938b-07a54f49cb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:01:43.407586 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.407558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:43.443503 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.443475 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.443503 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.443500 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408c80-9210-425d-938b-07a54f49cb40-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.443503 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.443510 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.443732 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.443518 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcks\" (UniqueName: \"kubernetes.io/projected/bc408c80-9210-425d-938b-07a54f49cb40-kube-api-access-5jcks\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.443732 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.443528 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408c80-9210-425d-938b-07a54f49cb40-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:01:43.965766 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.965731 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc408c80-9210-425d-938b-07a54f49cb40" containerID="fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603" exitCode=0 Jul 20 14:01:43.965916 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.965780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" Jul 20 14:01:43.965916 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.965807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerDied","Data":"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603"} Jul 20 14:01:43.965916 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.965856 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7" event={"ID":"bc408c80-9210-425d-938b-07a54f49cb40","Type":"ContainerDied","Data":"83281d50d952204e2ad59a2ee7dbb92104b5ef98ae73d88f8f0e1d056cf4e9ed"} Jul 20 14:01:43.965916 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.965878 2577 scope.go:117] "RemoveContainer" containerID="fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603" Jul 20 14:01:43.975313 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.975296 2577 scope.go:117] "RemoveContainer" containerID="df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed" Jul 20 14:01:43.983405 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.983389 2577 scope.go:117] "RemoveContainer" containerID="fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603" Jul 20 14:01:43.983658 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:01:43.983636 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603\": container with ID starting with fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603 not found: ID does not exist" containerID="fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603" Jul 20 14:01:43.983737 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.983669 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603"} err="failed to get container status \"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603\": rpc error: code = NotFound desc = could not find container \"fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603\": container with ID starting with fd6f296a532ba61d5cc97a4db27587033fbdec9bfa97638e942c570872f3a603 not found: ID does not exist" Jul 20 14:01:43.983737 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.983694 2577 scope.go:117] "RemoveContainer" containerID="df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed" Jul 20 14:01:43.983918 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:01:43.983902 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed\": container with ID starting with df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed not found: ID does not exist" containerID="df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed" Jul 20 14:01:43.983970 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.983926 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed"} err="failed to get container status \"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed\": rpc error: code = NotFound desc = could not find container \"df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed\": container with ID starting with df05730994e014a82dfc8ee49e4ea2a81c115c99b5b32c6ef347e6c48b62d9ed not found: ID does not exist" Jul 20 14:01:43.989606 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:43.989583 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:01:44.005620 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:44.005587 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ad282e-openshift-default-5956cb979c-82vg7"] Jul 20 14:01:45.394758 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:45.394716 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc408c80-9210-425d-938b-07a54f49cb40" path="/var/lib/kubelet/pods/bc408c80-9210-425d-938b-07a54f49cb40/volumes" Jul 20 14:01:46.863664 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:46.863627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:47.347278 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.347187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:47.430748 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.430652 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 14:01:47.439793 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.439770 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 14:01:47.446318 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.446300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 14:01:47.453155 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.453136 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 14:01:47.733269 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733240 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:01:47.733646 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733634 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f082b94f-6917-4b0f-b9af-8eb2a5d53426" containerName="revoke-keys" Jul 20 14:01:47.733695 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733648 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f082b94f-6917-4b0f-b9af-8eb2a5d53426" containerName="revoke-keys" Jul 20 14:01:47.733695 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733658 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:01:47.733695 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733663 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:01:47.733797 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733743 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f082b94f-6917-4b0f-b9af-8eb2a5d53426" containerName="revoke-keys" Jul 20 14:01:47.733797 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733752 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:01:47.733797 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.733759 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:01:47.738294 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.738276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.749143 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.749127 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-92659e-openshift-default-dockercfg-krsjp\"" Jul 20 14:01:47.763216 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.763194 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:01:47.794837 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.794809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:47.883761 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.883761 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7c4\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.883983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.884246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.884007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984558 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984558 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984558 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984834 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7c4\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984834 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984834 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984987 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984987 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984987 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.984987 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.984948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.985622 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.985129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.985622 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.985233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.985622 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.985321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.985622 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.985594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.987002 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.986981 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.987550 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.987531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.996857 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.996812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7c4\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:47.997286 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:47.997220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token\") pod \"e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:48.047960 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.047932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:48.161947 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.161910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:48.212280 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.212256 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:01:48.213054 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:01:48.213027 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaebc7e_a4f4_4f97_81d5_5bbce8b8a429.slice/crio-968ac4d0aede65f83cbd1fabb6b33749a75ce96883053d1eb5e23db08786c91b WatchSource:0}: Error finding container 968ac4d0aede65f83cbd1fabb6b33749a75ce96883053d1eb5e23db08786c91b: Status 404 returned error can't find the container with id 968ac4d0aede65f83cbd1fabb6b33749a75ce96883053d1eb5e23db08786c91b Jul 20 14:01:48.215188 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.215143 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:01:48.215256 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.215244 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:01:48.215293 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.215275 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:01:48.998462 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.998424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerStarted","Data":"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501"} Jul 20 14:01:48.998900 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:48.998481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerStarted","Data":"968ac4d0aede65f83cbd1fabb6b33749a75ce96883053d1eb5e23db08786c91b"} Jul 20 14:01:49.048525 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:49.048485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:49.060309 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:49.060275 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:49.099959 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:49.099903 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podStartSLOduration=2.099886652 podStartE2EDuration="2.099886652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:01:49.045688147 +0000 UTC m=+2708.110037064" watchObservedRunningTime="2026-07-20 14:01:49.099886652 +0000 UTC m=+2708.164235638" Jul 20 14:01:50.005777 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:50.005733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:50.006885 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:50.006855 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:01:51.075864 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:51.075830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:51.208587 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:51.208552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:51.344890 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:51.344798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:51.599939 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:51.599837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:52.112967 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:52.112926 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:52.231928 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:52.231900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:01:55.533994 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:01:55.533950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:03.084889 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.084852 2577 generic.go:356] "Generic (PLEG): container finished" podID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerID="8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501" exitCode=0 Jul 20 14:02:03.085321 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.084923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerDied","Data":"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501"} Jul 20 14:02:03.085416 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.085344 2577 scope.go:117] "RemoveContainer" containerID="8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501" Jul 20 14:02:03.086464 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.086439 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:02:03.086539 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.086518 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:02:03.086575 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:03.086554 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 14:02:04.094622 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:04.094580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerStarted","Data":"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c"} Jul 20 14:02:04.408415 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:04.408364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:04.504916 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:04.504886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:05.006110 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:05.006075 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:02:05.048681 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:05.048647 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:02:05.049878 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:05.049851 2577 patch_prober.go:28] interesting pod/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 20 14:02:05.049995 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:05.049900 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 20 14:02:06.049083 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:06.049046 2577 patch_prober.go:28] interesting pod/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 20 14:02:06.049535 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:06.049121 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 20 14:02:07.048898 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:07.048860 2577 patch_prober.go:28] interesting pod/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 20 14:02:07.049104 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:07.048929 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 20 14:02:07.707010 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:07.706975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:08.049472 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:08.049372 2577 patch_prober.go:28] interesting pod/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 20 14:02:08.049472 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:08.049441 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 20 14:02:09.052692 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:09.052659 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:02:09.053658 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:09.053638 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:02:10.677066 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:10.677030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:15.032541 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:15.032502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:15.285679 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:15.285578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:02:38.215592 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:02:38.215554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:03:39.686195 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:03:39.686149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:03:39.771557 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:03:39.771522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:04:41.940411 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:04:41.940323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:04:43.161011 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:04:43.160980 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:04:45.891981 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:04:45.891942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:15.374856 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:15.374813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:33.927616 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:33.927583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:34.095681 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:34.095649 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:56.162831 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:56.162792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:56.981463 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:56.981415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:05:59.371720 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:05:59.371680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:06:20.264693 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:20.264576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:06:33.937219 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:33.937161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:06:47.472646 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:47.472617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 14:06:47.479441 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:47.479416 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 14:06:47.488707 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:47.488686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 14:06:47.495392 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:06:47.495374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hddgc_8dec058e-f451-44de-a6a7-46f9725fd4c8/ovn-acl-logging/0.log" Jul 20 14:07:00.797206 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:00.797155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:01.461745 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:01.461701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:02.428902 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.428868 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178980400"] Jul 20 14:07:02.431237 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.429330 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:07:02.431237 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.429342 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc408c80-9210-425d-938b-07a54f49cb40" containerName="istio-proxy" Jul 20 14:07:02.432291 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.432276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:02.435861 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.435840 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6566r\"" Jul 20 14:07:02.441986 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.441959 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178980400"] Jul 20 14:07:02.481889 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.481850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7pn\" (UniqueName: \"kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn\") pod \"test-curl-7462-140138178980400\" (UID: \"d7b70495-eb95-4f39-947d-66da823dcce6\") " pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:02.582537 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.582501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7pn\" (UniqueName: \"kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn\") pod \"test-curl-7462-140138178980400\" (UID: \"d7b70495-eb95-4f39-947d-66da823dcce6\") " pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:02.591741 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.591711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7pn\" (UniqueName: \"kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn\") pod \"test-curl-7462-140138178980400\" (UID: \"d7b70495-eb95-4f39-947d-66da823dcce6\") " pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:02.742610 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.742520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:02.883534 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.883507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178980400"] Jul 20 14:07:02.884638 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:02.884610 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b70495_eb95_4f39_947d_66da823dcce6.slice/crio-f3057d84a18a07f433b6d57c64cbf9770a75eefc986ce82ed6212e8930e0537f WatchSource:0}: Error finding container f3057d84a18a07f433b6d57c64cbf9770a75eefc986ce82ed6212e8930e0537f: Status 404 returned error can't find the container with id f3057d84a18a07f433b6d57c64cbf9770a75eefc986ce82ed6212e8930e0537f Jul 20 14:07:02.886058 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:02.886039 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 14:07:03.864776 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:03.864747 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7b70495-eb95-4f39-947d-66da823dcce6" containerID="6e1bf0dcfa61aad79df269ea66a407ccef68613ac4abc91b43155d9ca7628ea0" exitCode=0 Jul 20 14:07:03.865160 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:03.864835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178980400" event={"ID":"d7b70495-eb95-4f39-947d-66da823dcce6","Type":"ContainerDied","Data":"6e1bf0dcfa61aad79df269ea66a407ccef68613ac4abc91b43155d9ca7628ea0"} Jul 20 14:07:03.865160 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:03.864887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178980400" event={"ID":"d7b70495-eb95-4f39-947d-66da823dcce6","Type":"ContainerStarted","Data":"f3057d84a18a07f433b6d57c64cbf9770a75eefc986ce82ed6212e8930e0537f"} Jul 20 14:07:05.010757 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.010731 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:05.061614 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.061585 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178980400_d7b70495-eb95-4f39-947d-66da823dcce6/test-curl-7462-140138178980400/0.log" Jul 20 14:07:05.100247 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.100214 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178980400"] Jul 20 14:07:05.105109 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.105088 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h7pn\" (UniqueName: \"kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn\") pod \"d7b70495-eb95-4f39-947d-66da823dcce6\" (UID: \"d7b70495-eb95-4f39-947d-66da823dcce6\") " Jul 20 14:07:05.105648 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.105625 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178980400"] Jul 20 14:07:05.107262 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.107231 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn" (OuterVolumeSpecName: "kube-api-access-6h7pn") pod "d7b70495-eb95-4f39-947d-66da823dcce6" (UID: "d7b70495-eb95-4f39-947d-66da823dcce6"). InnerVolumeSpecName "kube-api-access-6h7pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:05.206104 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.206032 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7pn\" (UniqueName: \"kubernetes.io/projected/d7b70495-eb95-4f39-947d-66da823dcce6-kube-api-access-6h7pn\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:05.230710 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.230682 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178980528"] Jul 20 14:07:05.231125 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.231112 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b70495-eb95-4f39-947d-66da823dcce6" containerName="test-curl-7462-140138178980400" Jul 20 14:07:05.231202 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.231126 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b70495-eb95-4f39-947d-66da823dcce6" containerName="test-curl-7462-140138178980400" Jul 20 14:07:05.231246 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.231213 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b70495-eb95-4f39-947d-66da823dcce6" containerName="test-curl-7462-140138178980400" Jul 20 14:07:05.234401 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.234378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:05.244503 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.244477 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178980528"] Jul 20 14:07:05.306617 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.306586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8j8\" (UniqueName: \"kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8\") pod \"test-curl-7462-140138178980528\" (UID: \"3e244eed-c761-47d4-83a9-a70ce47688f8\") " pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:05.394057 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.394021 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b70495-eb95-4f39-947d-66da823dcce6" path="/var/lib/kubelet/pods/d7b70495-eb95-4f39-947d-66da823dcce6/volumes" Jul 20 14:07:05.397588 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.397556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:05.407843 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.407819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8j8\" (UniqueName: \"kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8\") pod \"test-curl-7462-140138178980528\" (UID: \"3e244eed-c761-47d4-83a9-a70ce47688f8\") " pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:05.416276 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.416252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8j8\" (UniqueName: \"kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8\") pod \"test-curl-7462-140138178980528\" (UID: \"3e244eed-c761-47d4-83a9-a70ce47688f8\") " pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:05.498733 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.498656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:05.544052 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.544029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:05.708637 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.708417 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178980528"] Jul 20 14:07:05.711338 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:05.711310 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e244eed_c761_47d4_83a9_a70ce47688f8.slice/crio-03c163ba0b9e89046ff2154be80430b610c98a5c8c4ff90a0377d59b34404296 WatchSource:0}: Error finding container 03c163ba0b9e89046ff2154be80430b610c98a5c8c4ff90a0377d59b34404296: Status 404 returned error can't find the container with id 03c163ba0b9e89046ff2154be80430b610c98a5c8c4ff90a0377d59b34404296 Jul 20 14:07:05.877562 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.877536 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980400" Jul 20 14:07:05.877737 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.877595 2577 scope.go:117] "RemoveContainer" containerID="6e1bf0dcfa61aad79df269ea66a407ccef68613ac4abc91b43155d9ca7628ea0" Jul 20 14:07:05.878932 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:05.878908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178980528" event={"ID":"3e244eed-c761-47d4-83a9-a70ce47688f8","Type":"ContainerStarted","Data":"03c163ba0b9e89046ff2154be80430b610c98a5c8c4ff90a0377d59b34404296"} Jul 20 14:07:06.886183 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:06.886139 2577 generic.go:356] "Generic (PLEG): container finished" podID="3e244eed-c761-47d4-83a9-a70ce47688f8" containerID="aabfe5c39692c4c1478c55eb3e691b60a6da72142f80027f039303aa5f9b8b7b" exitCode=0 Jul 20 14:07:06.886647 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:06.886228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178980528" event={"ID":"3e244eed-c761-47d4-83a9-a70ce47688f8","Type":"ContainerDied","Data":"aabfe5c39692c4c1478c55eb3e691b60a6da72142f80027f039303aa5f9b8b7b"} Jul 20 14:07:08.024620 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.024596 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:08.045224 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.045160 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178980528_3e244eed-c761-47d4-83a9-a70ce47688f8/test-curl-7462-140138178980528/0.log" Jul 20 14:07:08.070498 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.070473 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178980528"] Jul 20 14:07:08.074793 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.074763 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178980528"] Jul 20 14:07:08.131614 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.131586 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8j8\" (UniqueName: \"kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8\") pod \"3e244eed-c761-47d4-83a9-a70ce47688f8\" (UID: \"3e244eed-c761-47d4-83a9-a70ce47688f8\") " Jul 20 14:07:08.133630 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.133597 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8" (OuterVolumeSpecName: "kube-api-access-hc8j8") pod "3e244eed-c761-47d4-83a9-a70ce47688f8" (UID: "3e244eed-c761-47d4-83a9-a70ce47688f8"). InnerVolumeSpecName "kube-api-access-hc8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:08.232448 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.232386 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8j8\" (UniqueName: \"kubernetes.io/projected/3e244eed-c761-47d4-83a9-a70ce47688f8-kube-api-access-hc8j8\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:08.233845 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.233825 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:08.234289 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.234266 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e244eed-c761-47d4-83a9-a70ce47688f8" containerName="test-curl-7462-140138178980528" Jul 20 14:07:08.234336 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.234290 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e244eed-c761-47d4-83a9-a70ce47688f8" containerName="test-curl-7462-140138178980528" Jul 20 14:07:08.234384 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.234375 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e244eed-c761-47d4-83a9-a70ce47688f8" containerName="test-curl-7462-140138178980528" Jul 20 14:07:08.237355 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.237339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:08.245319 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.245294 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:08.333114 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.333084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgg7v\" (UniqueName: \"kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v\") pod \"test-curl-7462-140138178904400\" (UID: \"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:08.433521 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.433498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgg7v\" (UniqueName: \"kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v\") pod \"test-curl-7462-140138178904400\" (UID: \"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:08.442834 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.442814 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgg7v\" (UniqueName: \"kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v\") pod \"test-curl-7462-140138178904400\" (UID: \"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:08.547489 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.547417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:08.686774 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.686744 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:08.688180 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:08.688135 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bfd7de3_7ad5_47da_8b75_34cc945e2fcb.slice/crio-734c5448d3cefa452aeb6d1b48fdb9e5b47cd2aa1b15effc697bb2f1cbe4ba9c WatchSource:0}: Error finding container 734c5448d3cefa452aeb6d1b48fdb9e5b47cd2aa1b15effc697bb2f1cbe4ba9c: Status 404 returned error can't find the container with id 734c5448d3cefa452aeb6d1b48fdb9e5b47cd2aa1b15effc697bb2f1cbe4ba9c Jul 20 14:07:08.903918 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.903889 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:08.903918 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.903918 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c163ba0b9e89046ff2154be80430b610c98a5c8c4ff90a0377d59b34404296" Jul 20 14:07:08.905396 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:08.905370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178904400" event={"ID":"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb","Type":"ContainerStarted","Data":"734c5448d3cefa452aeb6d1b48fdb9e5b47cd2aa1b15effc697bb2f1cbe4ba9c"} Jul 20 14:07:08.915689 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:07:08.915664 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-140138178980528\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="3e244eed-c761-47d4-83a9-a70ce47688f8" pod="opendatahub/test-curl-7462-140138178980528" Jul 20 14:07:09.395352 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:09.395318 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e244eed-c761-47d4-83a9-a70ce47688f8" path="/var/lib/kubelet/pods/3e244eed-c761-47d4-83a9-a70ce47688f8/volumes" Jul 20 14:07:09.912549 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:09.912497 2577 generic.go:356] "Generic (PLEG): container finished" podID="6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" containerID="8d4726ef5dcf88bc1bbf85272379a42c6a0d438d0b48a1c8191103206c4a570d" exitCode=0 Jul 20 14:07:09.912549 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:09.912549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178904400" event={"ID":"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb","Type":"ContainerDied","Data":"8d4726ef5dcf88bc1bbf85272379a42c6a0d438d0b48a1c8191103206c4a570d"} Jul 20 14:07:11.061738 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.061708 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.084085 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.084061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178904400_6bfd7de3-7ad5-47da-8b75-34cc945e2fcb/test-curl-7462-140138178904400/0.log" Jul 20 14:07:11.110133 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.110107 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:11.115879 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.115857 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:11.158598 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.158573 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgg7v\" (UniqueName: \"kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v\") pod \"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb\" (UID: \"6bfd7de3-7ad5-47da-8b75-34cc945e2fcb\") " Jul 20 14:07:11.160557 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.160530 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v" (OuterVolumeSpecName: "kube-api-access-jgg7v") pod "6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" (UID: "6bfd7de3-7ad5-47da-8b75-34cc945e2fcb"). InnerVolumeSpecName "kube-api-access-jgg7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:11.237207 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.237111 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:11.237624 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.237603 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" containerName="test-curl-7462-140138178904400" Jul 20 14:07:11.237721 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.237626 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" containerName="test-curl-7462-140138178904400" Jul 20 14:07:11.237774 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.237766 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" containerName="test-curl-7462-140138178904400" Jul 20 14:07:11.240970 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.240949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.246776 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.246756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:11.259606 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.259586 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgg7v\" (UniqueName: \"kubernetes.io/projected/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb-kube-api-access-jgg7v\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:11.360123 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.360092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8rj\" (UniqueName: \"kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj\") pod \"test-curl-7462-140138178904400\" (UID: \"1a8c64e3-eda6-402f-8736-1abad7d5b209\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.396237 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.396210 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfd7de3-7ad5-47da-8b75-34cc945e2fcb" path="/var/lib/kubelet/pods/6bfd7de3-7ad5-47da-8b75-34cc945e2fcb/volumes" Jul 20 14:07:11.461253 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.461220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8rj\" (UniqueName: \"kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj\") pod \"test-curl-7462-140138178904400\" (UID: \"1a8c64e3-eda6-402f-8736-1abad7d5b209\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.469985 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.469963 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8rj\" (UniqueName: \"kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj\") pod \"test-curl-7462-140138178904400\" (UID: \"1a8c64e3-eda6-402f-8736-1abad7d5b209\") " pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.551628 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.551551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.692425 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.692361 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:11.693711 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:11.693674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8c64e3_eda6_402f_8736_1abad7d5b209.slice/crio-a289f97bba41d042702a24827683ce817f61eabf2e70979c7ffddf9e7c063c64 WatchSource:0}: Error finding container a289f97bba41d042702a24827683ce817f61eabf2e70979c7ffddf9e7c063c64: Status 404 returned error can't find the container with id a289f97bba41d042702a24827683ce817f61eabf2e70979c7ffddf9e7c063c64 Jul 20 14:07:11.927443 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.927411 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178904400" event={"ID":"1a8c64e3-eda6-402f-8736-1abad7d5b209","Type":"ContainerStarted","Data":"a289f97bba41d042702a24827683ce817f61eabf2e70979c7ffddf9e7c063c64"} Jul 20 14:07:11.928571 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.928548 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:11.928571 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:11.928570 2577 scope.go:117] "RemoveContainer" containerID="8d4726ef5dcf88bc1bbf85272379a42c6a0d438d0b48a1c8191103206c4a570d" Jul 20 14:07:12.936954 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:12.936918 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a8c64e3-eda6-402f-8736-1abad7d5b209" containerID="071c24a3a8ecb18f430cfa67b8a7f595ac32e34f536af16cb611db5aaf603db4" exitCode=0 Jul 20 14:07:12.937379 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:12.937001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178904400" event={"ID":"1a8c64e3-eda6-402f-8736-1abad7d5b209","Type":"ContainerDied","Data":"071c24a3a8ecb18f430cfa67b8a7f595ac32e34f536af16cb611db5aaf603db4"} Jul 20 14:07:14.084253 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.084228 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:14.105908 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.105878 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178904400_1a8c64e3-eda6-402f-8736-1abad7d5b209/test-curl-7462-140138178904400/0.log" Jul 20 14:07:14.131367 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.131336 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:14.134816 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.134787 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178904400"] Jul 20 14:07:14.188454 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.188426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm8rj\" (UniqueName: \"kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj\") pod \"1a8c64e3-eda6-402f-8736-1abad7d5b209\" (UID: \"1a8c64e3-eda6-402f-8736-1abad7d5b209\") " Jul 20 14:07:14.190560 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.190527 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj" (OuterVolumeSpecName: "kube-api-access-gm8rj") pod "1a8c64e3-eda6-402f-8736-1abad7d5b209" (UID: "1a8c64e3-eda6-402f-8736-1abad7d5b209"). InnerVolumeSpecName "kube-api-access-gm8rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:14.267140 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.267110 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:14.267576 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.267558 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8c64e3-eda6-402f-8736-1abad7d5b209" containerName="test-curl-7462-140138178904400" Jul 20 14:07:14.267576 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.267573 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8c64e3-eda6-402f-8736-1abad7d5b209" containerName="test-curl-7462-140138178904400" Jul 20 14:07:14.267740 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.267654 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8c64e3-eda6-402f-8736-1abad7d5b209" containerName="test-curl-7462-140138178904400" Jul 20 14:07:14.270865 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.270847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:14.276919 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.276893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:14.289415 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.289391 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm8rj\" (UniqueName: \"kubernetes.io/projected/1a8c64e3-eda6-402f-8736-1abad7d5b209-kube-api-access-gm8rj\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:14.390556 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.390519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwt5\" (UniqueName: \"kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5\") pod \"test-curl-7462-140138178905408\" (UID: \"79926fa3-b8db-4137-905e-61cd5722ee67\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:14.491980 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.491887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwt5\" (UniqueName: \"kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5\") pod \"test-curl-7462-140138178905408\" (UID: \"79926fa3-b8db-4137-905e-61cd5722ee67\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:14.501276 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.501247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwt5\" (UniqueName: \"kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5\") pod \"test-curl-7462-140138178905408\" (UID: \"79926fa3-b8db-4137-905e-61cd5722ee67\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:14.581161 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.581126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:14.722584 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.722544 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:14.723602 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:14.723569 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79926fa3_b8db_4137_905e_61cd5722ee67.slice/crio-6f3af9bcb69e734ca3c6c1e6ef7718d3971ec66856ae08132d19cc3c7dcb0273 WatchSource:0}: Error finding container 6f3af9bcb69e734ca3c6c1e6ef7718d3971ec66856ae08132d19cc3c7dcb0273: Status 404 returned error can't find the container with id 6f3af9bcb69e734ca3c6c1e6ef7718d3971ec66856ae08132d19cc3c7dcb0273 Jul 20 14:07:14.951160 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.951124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178905408" event={"ID":"79926fa3-b8db-4137-905e-61cd5722ee67","Type":"ContainerStarted","Data":"6f3af9bcb69e734ca3c6c1e6ef7718d3971ec66856ae08132d19cc3c7dcb0273"} Jul 20 14:07:14.952557 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.952529 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a289f97bba41d042702a24827683ce817f61eabf2e70979c7ffddf9e7c063c64" Jul 20 14:07:14.952656 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:14.952561 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:14.963702 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:07:14.963676 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-140138178904400\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="1a8c64e3-eda6-402f-8736-1abad7d5b209" pod="opendatahub/test-curl-7462-140138178904400" Jul 20 14:07:15.394145 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:15.394109 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8c64e3-eda6-402f-8736-1abad7d5b209" path="/var/lib/kubelet/pods/1a8c64e3-eda6-402f-8736-1abad7d5b209/volumes" Jul 20 14:07:15.959751 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:15.959719 2577 generic.go:356] "Generic (PLEG): container finished" podID="79926fa3-b8db-4137-905e-61cd5722ee67" containerID="980d602b2ed4ebe5956b5d166d8aa6fe15cbc0ed6a1e45791f4bcc779ee830e2" exitCode=0 Jul 20 14:07:15.959943 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:15.959806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178905408" event={"ID":"79926fa3-b8db-4137-905e-61cd5722ee67","Type":"ContainerDied","Data":"980d602b2ed4ebe5956b5d166d8aa6fe15cbc0ed6a1e45791f4bcc779ee830e2"} Jul 20 14:07:16.221286 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:16.221204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:16.325600 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:16.325567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:17.102956 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.102932 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.125585 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.125554 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178905408_79926fa3-b8db-4137-905e-61cd5722ee67/test-curl-7462-140138178905408/0.log" Jul 20 14:07:17.159643 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.159615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:17.168597 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.168570 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:17.219062 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.218994 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hwt5\" (UniqueName: \"kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5\") pod \"79926fa3-b8db-4137-905e-61cd5722ee67\" (UID: \"79926fa3-b8db-4137-905e-61cd5722ee67\") " Jul 20 14:07:17.220966 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.220931 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5" (OuterVolumeSpecName: "kube-api-access-4hwt5") pod "79926fa3-b8db-4137-905e-61cd5722ee67" (UID: "79926fa3-b8db-4137-905e-61cd5722ee67"). InnerVolumeSpecName "kube-api-access-4hwt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:17.287618 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.287592 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:17.288074 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.288057 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79926fa3-b8db-4137-905e-61cd5722ee67" containerName="test-curl-7462-140138178905408" Jul 20 14:07:17.288163 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.288076 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79926fa3-b8db-4137-905e-61cd5722ee67" containerName="test-curl-7462-140138178905408" Jul 20 14:07:17.288270 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.288203 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79926fa3-b8db-4137-905e-61cd5722ee67" containerName="test-curl-7462-140138178905408" Jul 20 14:07:17.291387 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.291363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.300269 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.300248 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:17.320339 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.320316 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwt5\" (UniqueName: \"kubernetes.io/projected/79926fa3-b8db-4137-905e-61cd5722ee67-kube-api-access-4hwt5\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:17.394093 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.394055 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79926fa3-b8db-4137-905e-61cd5722ee67" path="/var/lib/kubelet/pods/79926fa3-b8db-4137-905e-61cd5722ee67/volumes" Jul 20 14:07:17.421659 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.421623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvct\" (UniqueName: \"kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct\") pod \"test-curl-7462-140138178905408\" (UID: \"833798dc-9923-4412-ab99-c2b109265126\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.523079 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.522991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvct\" (UniqueName: \"kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct\") pod \"test-curl-7462-140138178905408\" (UID: \"833798dc-9923-4412-ab99-c2b109265126\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.534111 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.534083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvct\" (UniqueName: \"kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct\") pod \"test-curl-7462-140138178905408\" (UID: \"833798dc-9923-4412-ab99-c2b109265126\") " pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.602472 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.602440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.757574 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.757537 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:17.759876 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:07:17.759847 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod833798dc_9923_4412_ab99_c2b109265126.slice/crio-844f9310204454d35515595e01dbfc8e41512930810a5645621e166761c1b491 WatchSource:0}: Error finding container 844f9310204454d35515595e01dbfc8e41512930810a5645621e166761c1b491: Status 404 returned error can't find the container with id 844f9310204454d35515595e01dbfc8e41512930810a5645621e166761c1b491 Jul 20 14:07:17.973064 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.973029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178905408" event={"ID":"833798dc-9923-4412-ab99-c2b109265126","Type":"ContainerStarted","Data":"844f9310204454d35515595e01dbfc8e41512930810a5645621e166761c1b491"} Jul 20 14:07:17.981427 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.981403 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:17.981427 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:17.981396 2577 scope.go:117] "RemoveContainer" containerID="980d602b2ed4ebe5956b5d166d8aa6fe15cbc0ed6a1e45791f4bcc779ee830e2" Jul 20 14:07:18.989216 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:18.989160 2577 generic.go:356] "Generic (PLEG): container finished" podID="833798dc-9923-4412-ab99-c2b109265126" containerID="476744e1601486610bb9b0e4fbc6220fba1a9a13b2fde165c0cf4347372f9e00" exitCode=0 Jul 20 14:07:18.989648 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:18.989225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-140138178905408" event={"ID":"833798dc-9923-4412-ab99-c2b109265126","Type":"ContainerDied","Data":"476744e1601486610bb9b0e4fbc6220fba1a9a13b2fde165c0cf4347372f9e00"} Jul 20 14:07:20.120358 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.120336 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:20.145783 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.145758 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-140138178905408_833798dc-9923-4412-ab99-c2b109265126/test-curl-7462-140138178905408/0.log" Jul 20 14:07:20.175982 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.175957 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:20.181517 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.181488 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-140138178905408"] Jul 20 14:07:20.249986 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.249919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpvct\" (UniqueName: \"kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct\") pod \"833798dc-9923-4412-ab99-c2b109265126\" (UID: \"833798dc-9923-4412-ab99-c2b109265126\") " Jul 20 14:07:20.251947 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.251921 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct" (OuterVolumeSpecName: "kube-api-access-bpvct") pod "833798dc-9923-4412-ab99-c2b109265126" (UID: "833798dc-9923-4412-ab99-c2b109265126"). InnerVolumeSpecName "kube-api-access-bpvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:20.351495 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.351468 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvct\" (UniqueName: \"kubernetes.io/projected/833798dc-9923-4412-ab99-c2b109265126-kube-api-access-bpvct\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:07:20.976879 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:20.976845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:21.003918 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:21.003892 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:21.003918 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:21.003923 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="844f9310204454d35515595e01dbfc8e41512930810a5645621e166761c1b491" Jul 20 14:07:21.013955 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:07:21.013927 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-140138178905408\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="833798dc-9923-4412-ab99-c2b109265126" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:21.362163 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:21.362116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:21.393655 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:07:21.393629 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-140138178905408\" is forbidden: User \"system:node:ip-10-0-141-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-54.ec2.internal' and this object" podUID="833798dc-9923-4412-ab99-c2b109265126" pod="opendatahub/test-curl-7462-140138178905408" Jul 20 14:07:21.396201 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:21.396161 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833798dc-9923-4412-ab99-c2b109265126" path="/var/lib/kubelet/pods/833798dc-9923-4412-ab99-c2b109265126/volumes" Jul 20 14:07:21.887187 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:21.887130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:31.044385 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:31.044342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:35.467679 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:35.467593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:35.888220 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:35.888162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:41.722587 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:41.722546 2577 scope.go:117] "RemoveContainer" containerID="ba3b68d2e7558640c480121a0ba0ee4aa7f34f75c3660d6f910f17c4ffac9b2b" Jul 20 14:07:42.502962 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:42.502924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:42.615547 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:42.615511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:42.822837 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:42.822746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:43.099076 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:43.098984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:43.993133 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:43.993090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:44.093753 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:44.093716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:53.774939 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:53.774897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:58.304055 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:58.304011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:07:58.702385 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:07:58.702348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:05.269334 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.269298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:05.350939 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.350898 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:05.550714 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.550630 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:08:05.550905 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.550845 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" containerID="cri-o://76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c" gracePeriod=30 Jul 20 14:08:05.582135 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.582101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:05.803283 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:05.803197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:07.940196 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:07.940136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:08.038940 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:08.038904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:09.055841 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:09.055808 2577 patch_prober.go:28] interesting pod/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 14:08:09.056256 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:09.055865 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 14:08:10.812930 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.812863 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:08:10.822488 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822460 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822488 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822490 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822515 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt7c4\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822591 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822956 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822690 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822956 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs\") pod \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\" (UID: \"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429\") " Jul 20 14:08:10.822956 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822787 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:08:10.823120 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.822839 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:08:10.823191 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.823140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:08:10.823299 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.823271 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:08:10.823719 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.823694 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:08:10.825032 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.824945 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:08:10.825121 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.825041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:08:10.825121 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.825095 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4" (OuterVolumeSpecName: "kube-api-access-jt7c4") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "kube-api-access-jt7c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:08:10.825121 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.825108 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" (UID: "dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:08:10.923791 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923753 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-certs\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.923791 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923784 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-token\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.923791 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923796 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-credential-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923808 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7c4\" (UniqueName: \"kubernetes.io/projected/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-kube-api-access-jt7c4\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923820 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-workload-socket\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923833 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istiod-ca-cert\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923844 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-data\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923854 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-envoy\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:10.924031 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:10.923864 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429-istio-podinfo\") on node \"ip-10-0-141-54.ec2.internal\" DevicePath \"\"" Jul 20 14:08:11.315963 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.315930 2577 generic.go:356] "Generic (PLEG): container finished" podID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerID="76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c" exitCode=0 Jul 20 14:08:11.316135 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.315977 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" Jul 20 14:08:11.316135 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.316009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerDied","Data":"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c"} Jul 20 14:08:11.316135 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.316088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg" event={"ID":"dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429","Type":"ContainerDied","Data":"968ac4d0aede65f83cbd1fabb6b33749a75ce96883053d1eb5e23db08786c91b"} Jul 20 14:08:11.316135 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.316111 2577 scope.go:117] "RemoveContainer" containerID="76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c" Jul 20 14:08:11.326376 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.326360 2577 scope.go:117] "RemoveContainer" containerID="8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501" Jul 20 14:08:11.335015 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.334995 2577 scope.go:117] "RemoveContainer" containerID="76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c" Jul 20 14:08:11.335316 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:08:11.335297 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c\": container with ID starting with 76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c not found: ID does not exist" containerID="76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c" Jul 20 14:08:11.335373 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.335325 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c"} err="failed to get container status \"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c\": rpc error: code = NotFound desc = could not find container \"76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c\": container with ID starting with 76c26a1a8845c910b1067f47d811896fc3bcb35f4930d0413abebf0538ecec5c not found: ID does not exist" Jul 20 14:08:11.335373 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.335343 2577 scope.go:117] "RemoveContainer" containerID="8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501" Jul 20 14:08:11.335565 ip-10-0-141-54 kubenswrapper[2577]: E0720 14:08:11.335548 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501\": container with ID starting with 8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501 not found: ID does not exist" containerID="8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501" Jul 20 14:08:11.335609 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.335572 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501"} err="failed to get container status \"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501\": rpc error: code = NotFound desc = could not find container \"8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501\": container with ID starting with 8991bd567e69b5a46b90a3933f93aa48a941964e521109cfb3fab75379505501 not found: ID does not exist" Jul 20 14:08:11.342648 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.342626 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:08:11.348552 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.348530 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-92659e-openshift-default-9f7dc89c5-s6lfg"] Jul 20 14:08:11.393519 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:11.393495 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" path="/var/lib/kubelet/pods/dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429/volumes" Jul 20 14:08:12.140628 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:12.140602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d7vqj_cb4adb05-d8d5-4fad-89ed-785b7a36e3c0/manager/0.log" Jul 20 14:08:12.276328 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:12.276294 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xbz8c_a997532e-f76a-49b2-8116-79f131379202/manager/0.log" Jul 20 14:08:12.686337 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:12.686306 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mpzl2_9c771701-0b5e-46c5-ade8-6fca67cdec69/manager/0.log" Jul 20 14:08:12.923602 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:12.923576 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-s7hvp_9e444982-5266-4908-9b51-6069e40d502c/copy-manifests/0.log" Jul 20 14:08:12.934214 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:12.934184 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-s7hvp_9e444982-5266-4908-9b51-6069e40d502c/manager/0.log" Jul 20 14:08:13.774797 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:13.774764 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qmz4l_d5dd00ef-7212-4679-a886-24b79cbee889/postgres/0.log" Jul 20 14:08:14.610288 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.610239 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/util/0.log" Jul 20 14:08:14.617805 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.617783 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/pull/0.log" Jul 20 14:08:14.627514 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.627488 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/extract/0.log" Jul 20 14:08:14.755880 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.755849 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/util/0.log" Jul 20 14:08:14.765227 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.765196 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/pull/0.log" Jul 20 14:08:14.772889 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.772870 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/extract/0.log" Jul 20 14:08:14.886585 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.886505 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/util/0.log" Jul 20 14:08:14.900377 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.900346 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/pull/0.log" Jul 20 14:08:14.913828 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:14.913802 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/extract/0.log" Jul 20 14:08:15.041018 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.040987 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/extract/0.log" Jul 20 14:08:15.052614 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.052590 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/util/0.log" Jul 20 14:08:15.061298 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.061268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/pull/0.log" Jul 20 14:08:15.311442 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.311344 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tgtzh_f3ecd588-9f66-44f8-8644-55ea36a7ab61/manager/0.log" Jul 20 14:08:15.430523 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.430494 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-k778t_e78e390f-ad30-4a70-9909-16e614d07cb6/manager/0.log" Jul 20 14:08:15.540304 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.540274 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9scvc_f18f9562-fe7e-422a-a77f-b48868ad8989/kuadrant-console-plugin/0.log" Jul 20 14:08:15.663532 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.663506 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7t575_3d01e344-ed7d-4e7d-a28c-3b545b09b42c/registry-server/0.log" Jul 20 14:08:15.793894 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:15.793829 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8qpqb_1de09fb1-8af3-45c7-96ab-626d6f6f485e/manager/0.log" Jul 20 14:08:16.048972 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:16.048894 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ccwxs_c8b6ba05-4af1-46f3-a522-f899de075c31/manager/0.log" Jul 20 14:08:16.759297 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:16.759265 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-l6j7q_37fe0dae-c906-4370-87ad-da63eb63a6f5/kube-auth-proxy/0.log" Jul 20 14:08:16.895095 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:16.895060 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/1.log" Jul 20 14:08:17.020590 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:17.020501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pkj5z_f6fdefa1-5265-4ac0-82a0-53db3c1d8955/payload-pre-processing/0.log" Jul 20 14:08:17.148304 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:17.148269 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sh8z4_489ec3f6-112d-47fd-b801-2cf3667c1a9f/payload-processing/0.log" Jul 20 14:08:18.023742 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:18.023692 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frlxv7_116c30bf-454e-4905-ac6b-850b8666b348/main/0.log" Jul 20 14:08:18.139838 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:18.139806 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jnsvf_b88d16f3-9f28-4108-8eeb-c324fe48fae2/main/0.log" Jul 20 14:08:24.148699 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.148651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:24.249864 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.249830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" limit=1024 Jul 20 14:08:24.285222 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285191 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp56k/must-gather-lwk2c"] Jul 20 14:08:24.285637 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285624 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.285680 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285640 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.285680 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285655 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833798dc-9923-4412-ab99-c2b109265126" containerName="test-curl-7462-140138178905408" Jul 20 14:08:24.285680 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285662 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="833798dc-9923-4412-ab99-c2b109265126" containerName="test-curl-7462-140138178905408" Jul 20 14:08:24.285780 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285757 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.285780 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285767 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.285780 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285774 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="833798dc-9923-4412-ab99-c2b109265126" containerName="test-curl-7462-140138178905408" Jul 20 14:08:24.286125 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285860 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.286125 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.285867 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaebc7e-a4f4-4f97-81d5-5bbce8b8a429" containerName="istio-proxy" Jul 20 14:08:24.288938 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.288923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.292452 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.292433 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lp56k\"/\"default-dockercfg-m47wx\"" Jul 20 14:08:24.292939 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.292925 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"openshift-service-ca.crt\"" Jul 20 14:08:24.293309 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.293293 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"kube-root-ca.crt\"" Jul 20 14:08:24.303548 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.303521 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/must-gather-lwk2c"] Jul 20 14:08:24.343279 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.343247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/832cbd6e-c648-46a8-aec6-0bbe33811ba8-must-gather-output\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.343428 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.343356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrhf\" (UniqueName: \"kubernetes.io/projected/832cbd6e-c648-46a8-aec6-0bbe33811ba8-kube-api-access-bbrhf\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.444660 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.444564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrhf\" (UniqueName: \"kubernetes.io/projected/832cbd6e-c648-46a8-aec6-0bbe33811ba8-kube-api-access-bbrhf\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.444838 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.444661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/832cbd6e-c648-46a8-aec6-0bbe33811ba8-must-gather-output\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.445063 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.445036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/832cbd6e-c648-46a8-aec6-0bbe33811ba8-must-gather-output\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.460776 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.460750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrhf\" (UniqueName: \"kubernetes.io/projected/832cbd6e-c648-46a8-aec6-0bbe33811ba8-kube-api-access-bbrhf\") pod \"must-gather-lwk2c\" (UID: \"832cbd6e-c648-46a8-aec6-0bbe33811ba8\") " pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.598666 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.598632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/must-gather-lwk2c" Jul 20 14:08:24.750981 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:24.750948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/must-gather-lwk2c"] Jul 20 14:08:24.752448 ip-10-0-141-54 kubenswrapper[2577]: W0720 14:08:24.752419 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod832cbd6e_c648_46a8_aec6_0bbe33811ba8.slice/crio-7bc664923cce5d495323ed37ee5478e1114a1de00dda765d720c85dd3d2e9102 WatchSource:0}: Error finding container 7bc664923cce5d495323ed37ee5478e1114a1de00dda765d720c85dd3d2e9102: Status 404 returned error can't find the container with id 7bc664923cce5d495323ed37ee5478e1114a1de00dda765d720c85dd3d2e9102 Jul 20 14:08:25.400483 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:25.400447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/must-gather-lwk2c" event={"ID":"832cbd6e-c648-46a8-aec6-0bbe33811ba8","Type":"ContainerStarted","Data":"7bc664923cce5d495323ed37ee5478e1114a1de00dda765d720c85dd3d2e9102"} Jul 20 14:08:26.411909 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:26.411864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/must-gather-lwk2c" event={"ID":"832cbd6e-c648-46a8-aec6-0bbe33811ba8","Type":"ContainerStarted","Data":"a31ac51668817947b89b2f897798358570c68a193297808fb9ba951673054e80"} Jul 20 14:08:26.412400 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:26.411921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/must-gather-lwk2c" event={"ID":"832cbd6e-c648-46a8-aec6-0bbe33811ba8","Type":"ContainerStarted","Data":"d805f88b1772ea18d47ad9dd2210434f3584417771635a3f88f5c42f380e2008"} Jul 20 14:08:26.432067 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:26.432001 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp56k/must-gather-lwk2c" podStartSLOduration=1.5568939240000002 podStartE2EDuration="2.431978543s" totalImagesPullingTime="875.084619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:08:26.431925583 +0000 UTC m=+3105.496274500" watchObservedRunningTime="2026-07-20 14:08:26.431978543 +0000 UTC m=+3105.496327461" Jul 20 14:08:27.416954 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:27.416924 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mtlhw_6ed66974-03a7-40cc-9cea-65fc2a99cd57/global-pull-secret-syncer/0.log" Jul 20 14:08:27.540859 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:27.540819 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5wbzb_6e949be8-73ac-4b83-b1b6-0a1ace341746/konnectivity-agent/0.log" Jul 20 14:08:27.687587 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:27.687490 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-54.ec2.internal_73f6bdbf997a544246c875c3b1e88545/haproxy/0.log" Jul 20 14:08:32.629003 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:32.628967 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/extract/0.log" Jul 20 14:08:32.785808 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:32.785777 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/util/0.log" Jul 20 14:08:33.000332 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.000247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jbj5m_e1740153-04ec-470f-8901-d5b982e9f38d/pull/0.log" Jul 20 14:08:33.128195 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.128136 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/extract/0.log" Jul 20 14:08:33.314287 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.314210 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/util/0.log" Jul 20 14:08:33.532914 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.532880 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0x5rss_24830256-b72b-430c-8747-3eba634d1d51/pull/0.log" Jul 20 14:08:33.651833 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.651802 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/extract/0.log" Jul 20 14:08:33.755686 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.755651 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/util/0.log" Jul 20 14:08:33.829641 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.829610 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lnj8z_8f143e2b-3d07-4158-b7c1-d28fe54db4f9/pull/0.log" Jul 20 14:08:33.934998 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:33.934919 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/extract/0.log" Jul 20 14:08:34.011672 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:34.011643 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/util/0.log" Jul 20 14:08:34.120802 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:34.120763 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1pn4t4_52115695-a382-4a33-bd51-6ac29a7510d5/pull/0.log" Jul 20 14:08:34.898192 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:34.898141 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tgtzh_f3ecd588-9f66-44f8-8644-55ea36a7ab61/manager/0.log" Jul 20 14:08:34.979094 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:34.979061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-k778t_e78e390f-ad30-4a70-9909-16e614d07cb6/manager/0.log" Jul 20 14:08:35.065634 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:35.065612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9scvc_f18f9562-fe7e-422a-a77f-b48868ad8989/kuadrant-console-plugin/0.log" Jul 20 14:08:35.161417 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:35.161338 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7t575_3d01e344-ed7d-4e7d-a28c-3b545b09b42c/registry-server/0.log" Jul 20 14:08:35.488505 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:35.488423 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8qpqb_1de09fb1-8af3-45c7-96ab-626d6f6f485e/manager/0.log" Jul 20 14:08:35.690262 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:35.690231 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ccwxs_c8b6ba05-4af1-46f3-a522-f899de075c31/manager/0.log" Jul 20 14:08:37.059969 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.059931 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr"] Jul 20 14:08:37.075488 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.075454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.103634 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.103412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr"] Jul 20 14:08:37.177576 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.177529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-proc\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.177576 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.177578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-lib-modules\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.177822 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.177603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-podres\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.177822 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.177673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-sys\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.177822 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.177689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgdk\" (UniqueName: \"kubernetes.io/projected/0a0861dc-edd2-4de2-b90d-32a9b2855b32-kube-api-access-swgdk\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.278908 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.278868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-lib-modules\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279092 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.278982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-podres\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279092 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-lib-modules\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279092 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-sys\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279280 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgdk\" (UniqueName: \"kubernetes.io/projected/0a0861dc-edd2-4de2-b90d-32a9b2855b32-kube-api-access-swgdk\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279280 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-sys\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279280 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-podres\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279280 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-proc\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.279465 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.279386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a0861dc-edd2-4de2-b90d-32a9b2855b32-proc\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.294202 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.294158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgdk\" (UniqueName: \"kubernetes.io/projected/0a0861dc-edd2-4de2-b90d-32a9b2855b32-kube-api-access-swgdk\") pod \"perf-node-gather-daemonset-w9jrr\" (UID: \"0a0861dc-edd2-4de2-b90d-32a9b2855b32\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.338629 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.338533 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-55jnt_e29a4494-fce7-40c5-8324-20f42b671348/cluster-monitoring-operator/0.log" Jul 20 14:08:37.372128 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.372098 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-j98nd_89fad513-93c8-48b8-9cb7-df45b04bb0e3/kube-state-metrics/0.log" Jul 20 14:08:37.391639 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.391610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:37.412189 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.412150 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-j98nd_89fad513-93c8-48b8-9cb7-df45b04bb0e3/kube-rbac-proxy-main/0.log" Jul 20 14:08:37.461516 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.461427 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-j98nd_89fad513-93c8-48b8-9cb7-df45b04bb0e3/kube-rbac-proxy-self/0.log" Jul 20 14:08:37.528501 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.528430 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7dfdcb75f-45zq8_060566f1-4344-483e-b040-edddc62d9a04/metrics-server/0.log" Jul 20 14:08:37.580071 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.580038 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr"] Jul 20 14:08:37.590022 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.589952 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-5wvcx_40583dc8-d7f3-4536-8782-34f8e48f2bba/monitoring-plugin/0.log" Jul 20 14:08:37.803141 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.803109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntqfq_9e35aaf1-cf04-4796-b4f5-b098d2d9f66c/node-exporter/0.log" Jul 20 14:08:37.836204 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.836149 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntqfq_9e35aaf1-cf04-4796-b4f5-b098d2d9f66c/kube-rbac-proxy/0.log" Jul 20 14:08:37.868551 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:37.868520 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ntqfq_9e35aaf1-cf04-4796-b4f5-b098d2d9f66c/init-textfile/0.log" Jul 20 14:08:38.528376 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:38.528335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" event={"ID":"0a0861dc-edd2-4de2-b90d-32a9b2855b32","Type":"ContainerStarted","Data":"56a97e8b6600f047a561c35768e7ad2adadaa31196f8e49521cd983d344f3458"} Jul 20 14:08:38.528899 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:38.528393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" event={"ID":"0a0861dc-edd2-4de2-b90d-32a9b2855b32","Type":"ContainerStarted","Data":"4607097c8ab70997cbda1d90bbc168a719e10382c8a1a651b1db63f7e34a06a5"} Jul 20 14:08:38.528899 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:38.528482 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:38.565093 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:38.565044 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" podStartSLOduration=1.5650277670000001 podStartE2EDuration="1.565027767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 14:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:08:38.564089241 +0000 UTC m=+3117.628438156" watchObservedRunningTime="2026-07-20 14:08:38.565027767 +0000 UTC m=+3117.629376681" Jul 20 14:08:40.915596 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:40.915558 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/1.log" Jul 20 14:08:40.924544 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:40.924515 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-b2bdl_195a9890-4dc6-48ed-bde8-592fbfb9bf1e/console-operator/2.log" Jul 20 14:08:41.666837 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:41.666796 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-89sfj_54a75e55-a4bc-4576-b5a7-34de68eb6ec8/download-server/0.log" Jul 20 14:08:42.365357 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:42.365323 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-tx2lp_66987fa4-82c7-4811-8044-124065084704/volume-data-source-validator/0.log" Jul 20 14:08:43.653913 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:43.653883 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fw6d8_de2732cd-a004-4b76-8279-e95796aeea5f/dns/0.log" Jul 20 14:08:43.708093 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:43.708058 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fw6d8_de2732cd-a004-4b76-8279-e95796aeea5f/kube-rbac-proxy/0.log" Jul 20 14:08:43.892624 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:43.892594 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxjhc_2d220c08-dc87-46d4-a868-b10a521c3b40/dns-node-resolver/0.log" Jul 20 14:08:44.552970 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:44.552935 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-w9jrr" Jul 20 14:08:44.629713 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:44.629677 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c9d4cb47f-sm529_d0e92ed2-8df0-44b3-ab24-b9eee7e0b7c4/registry/0.log" Jul 20 14:08:44.670681 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:44.670645 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9kkq8_9c815171-5cd5-4db5-b95a-dc9395fb6000/node-ca/0.log" Jul 20 14:08:46.057362 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:46.057229 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-l6j7q_37fe0dae-c906-4370-87ad-da63eb63a6f5/kube-auth-proxy/0.log" Jul 20 14:08:46.094291 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:46.094255 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/0.log" Jul 20 14:08:46.145448 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:46.145408 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/1.log" Jul 20 14:08:46.199699 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:46.199653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pkj5z_f6fdefa1-5265-4ac0-82a0-53db3c1d8955/payload-pre-processing/0.log" Jul 20 14:08:46.243044 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:46.243013 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sh8z4_489ec3f6-112d-47fd-b801-2cf3667c1a9f/payload-processing/0.log" Jul 20 14:08:47.256433 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.256394 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2lqzk_ae1b7005-8291-4dec-ae98-0f038dc793dc/serve-healthcheck-canary/0.log" Jul 20 14:08:47.722553 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.722471 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-l6j7q_37fe0dae-c906-4370-87ad-da63eb63a6f5/kube-auth-proxy/0.log" Jul 20 14:08:47.748596 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.748565 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/0.log" Jul 20 14:08:47.799806 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.799772 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/1.log" Jul 20 14:08:47.831403 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.831372 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pkj5z_f6fdefa1-5265-4ac0-82a0-53db3c1d8955/payload-pre-processing/0.log" Jul 20 14:08:47.859814 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.859788 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sh8z4_489ec3f6-112d-47fd-b801-2cf3667c1a9f/payload-processing/0.log" Jul 20 14:08:47.943353 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.943321 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-bj7gj_66e6e7e3-929b-4928-9e5b-d3ac05572fad/insights-operator/0.log" Jul 20 14:08:47.943569 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:47.943549 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-bj7gj_66e6e7e3-929b-4928-9e5b-d3ac05572fad/insights-operator/1.log" Jul 20 14:08:48.041528 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:48.041454 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj7mq_1c86b05b-f379-4376-9275-0d09eb3be0a2/kube-rbac-proxy/0.log" Jul 20 14:08:48.068687 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:48.068658 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj7mq_1c86b05b-f379-4376-9275-0d09eb3be0a2/exporter/0.log" Jul 20 14:08:48.095137 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:48.095105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj7mq_1c86b05b-f379-4376-9275-0d09eb3be0a2/extractor/0.log" Jul 20 14:08:49.248100 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:49.248070 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-l6j7q_37fe0dae-c906-4370-87ad-da63eb63a6f5/kube-auth-proxy/0.log" Jul 20 14:08:49.270389 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:49.270351 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/0.log" Jul 20 14:08:49.323367 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:49.323327 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l4rvx_086e3d2c-e7d0-40dc-95d7-de597c80f1a6/istio-proxy/1.log" Jul 20 14:08:49.366090 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:49.366061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pkj5z_f6fdefa1-5265-4ac0-82a0-53db3c1d8955/payload-pre-processing/0.log" Jul 20 14:08:49.423872 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:49.423802 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sh8z4_489ec3f6-112d-47fd-b801-2cf3667c1a9f/payload-processing/0.log" Jul 20 14:08:50.688411 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:50.688376 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d7vqj_cb4adb05-d8d5-4fad-89ed-785b7a36e3c0/manager/0.log" Jul 20 14:08:51.055583 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:51.055504 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xbz8c_a997532e-f76a-49b2-8116-79f131379202/manager/0.log" Jul 20 14:08:51.627848 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:51.627816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mpzl2_9c771701-0b5e-46c5-ade8-6fca67cdec69/manager/0.log" Jul 20 14:08:51.892107 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:51.892076 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-s7hvp_9e444982-5266-4908-9b51-6069e40d502c/manager/0.log" Jul 20 14:08:51.936892 ip-10-0-141-54 kubenswrapper[2577]: I0720 14:08:51.936873 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-s7hvp_9e444982-5266-4908-9b51-6069e40d502c/copy-manifests/0.log"